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SANTRAUKA 

Straipsnyje analizuojama potencialiq vertybiij ir konfliktq jverti- 
nimo kriterijq strukturine klasifikacija. Tyrimo pagrindq sudaro 
konfliktLj tipq isskyrimas (tikslq ir priemoniLi konfliktai; kokybi- 
niai ir kiekybiniai konfliktai). Aptariamos sesios vertybiq sritys: 
etines, metodines, socialines ir interaktyviosios, taip pat istaty- 
mines, ekonomines ir asmenines. KonfliktLj, atsirandanciq tarp 
sip vertybiij strukturine klasifikacija taikyta konkreciai jvertinant 
destyma, kurj atliko Zalcburgo universiteto studentai. Aptariami 
kai kurie konfliktai, pvz., apreptis pries patikimumq; individua- 
lios pries socialines ir j kriterijus orientuotas standartines nor- 
mas; studentq reitingavimas pries jvertinimq, destytojq jtakaty- 
rimij rezultatams. Strukturinq klasifikacija siuloma taikyti pries 
atliekant jvertinimq. 

PAGRINDINiy SAVOKy APIBREZIMAI 

• Antagonizmas- priestaringq principq, jegp ar veiksnip santy- 
kis; siuo atveju priestaringi tikslai, nesuderinamos priemones. 

• Apreptis- siuo atveju sricip, vertinamp taikant duotqsias prie- 
mones, skaicius. 

• Normos, orientuotos / kriterijus -tokios normos, kai jvertinant 
studentus remiamasi is anksto nustatytais kriterijais, t. y., atsi- 
zvelgiama j tikslp pasiekimo laipsnj. 

• Sprendimp priemimas sudetingomis sociaiinemis apiinkybe- 
mis- diskusijos ir pasikeitimas argumentais uz ir/ar pries nesu- 
tampancias vertybes atskirp asmenp sqveikos ir komunikacijos 
procese. 

• Sprendimo priemimo apiinkybes - s\\uac\\a, kurioje protago- 
nistas prasomas priimti sprendimq del vieno ar daugiau veiks- 
mp. 

• Dilema -tok\a sprendimo priemimo situacija, kurioje bet koks 
priimtas sprendimas pazeis kokias nors kitas vertybes; papras- 
tai tai kokybine konfliktine situacija. 

• /vertinimo di/emos- sudetingos sprendimo priemimo aplinky- 
bes jvertinimo procese visuose svietimo sistemos sektoriuose. 

• /vertinimo procesas (suaugusiuju) svietime- sistemingas fak- 
tp ir kitos informacijos rinkimas bei interpretavimas siekiant pri- 
imti vertybinj sprendimo apie veiklos pobudj visose (suaugusip- 
jp) svietimo srityse. 

• Saziningumas - siuo atveju protagonistas turi progq laikytis 
jvertinimo kriterijp reikalavimp (t. y. laiku suzino tuos kriterijus ir 
turi galimybg jp siekti, pvz., jam destomas tam tikras kursas, 
kurio jsisavinimas veliau bus jvertintas). 


1 Is dalies remiamasi projektu Jvertinimas svietime 11 , grantas 

#P13594-SOZ, gautas is Austrijos mokslo fondo (FWF). 


ABSTRACT 

A taxonomy of potential values conflicts in evaluations is 
presented. Basis is the distinction of types of conflict (aims 
vs. means conflicts; qualitative vs. quantitative conflicts). Six 
areas of values are discussed: the ethical, methodical, social 
and interactive, legal, economic, and personal values. The 
taxonomy of conflicts between these values is applied to a 
concrete evaluation process, namely the student rating of 
teaching at the University of Salzburg. Some of the conflicts 
are discussed, e.g. bandwidth vs. fidelity, individual vs. social 
and criterion oriented reference norm, student ratings vs. 
evaluation, and cheating teachers. It is suggested to use the 
taxonomy before starting an evaluation. 

DEFINITIONS OF KEY WORDS 

• Antagonism : relation between opposing principles or forces 
or factor; in this particular case: opposed goals, incompatible 
means. 

• Bandwidth : here number of domains that are assessed with 
a given instrument. 

• Criterion oriented norm : reference norm to evaluate stu- 
dents based on pre-determined criteria, i.e., referring to the 
degree of accomplishment of goals. 

• Decision-making in difficult social-situations, discussion and 
interchange of arguments concerning the pro and / or contra 
for values, which conflict in the interaction and communication 
of persons. 

• Decision situation-, situation in which the protagonist is as- 
ked to take a decision between two or more actions. 

• Dilemma-, decision situation in which any decision will violate 
some values; usually a qualitative conflict situation. 

• Dilemmas in evaluations, (difficult) decision situations in eva- 
luation processes of all possible sections concerning education. 

• Evaluation-process in (adult) education : systematic collec- 
tion and interpretation of facts and hints with the goal to 
achieve a decision of values about actions in all fields of (adult) 
education. 

• Fairness : here the protagonist gets the opportunity to fulfill 
the requirements with respect to the criteria which are used 
for the evaluation (i.e., knows the criteria in time and has the 
possibility to achieve them, e.g., is taught the content which 
is evaluated later on). 


1 Partly based on the project “Evaluation in Education’’, grant 
# P13594-SOZ by the Austrian Science Fund (FWF). 
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* Patikimumas- siuo atveju tikslumas, vertinant savybq, taikant 
duotqjq priemonq. 

* Formuojamasis jvertinimas -\ver\\n\mas pasitelkiamas siekiant 
daryti jtakq analizuojamam procesui. 

* /vertinimo / svietimo tikslai - jie apibudina tai, ko kas nors 
siekia duotomis aplinkybemis. 

* Priemones jvertinime / svietime - apibudina, kq galima daryti 
norint pasiekti tiksla. duotomis aplinkybemis. 

* Kokybine konfliktine situacija - tokia sprendimo priemimo si- 
tuacija, kai galimi veiksmai ar vertybiq pasiekimas yra nesuderi- 
nami ( arba-arba ). 

* Kiekybine konfliktine situacija - tokia sprendimo priemimo si- 
tuacija, kai galimi veiksmai gali buti apibudinami sqvokomis dau- 
giau ar maziau; kuo labiau laikomasi vienos vertybes, tuo ma- 
ziau lieka vietos kitai. 

* Atsakomybe- (1 ) jega, vercianti jus iaikytis jsipareigojimu; (2) 
veiksmq eiga, kurios ta jega reikalauja, jai esant isorinei (pvz., 
kitij zmoniq nustatytos normq sistemos) ar vidinei (pvz., prota- 
gonisto sqzine paremtos normq sistemos). 

* Atiiepiamas jvertinimas (mode/is) - svarbios atliepiamojo jver- 
tinimo proceso vertybes yra (1) skaidrumas apie visus jvertini- 
mo zingsnius, (2) visq j vertinimo procesq jtrauktq asmenq daly- 
vavimas, pvz., dalinai priimantsvarbius jvertinimo proceso spren- 
dimus ir (3) tolesnes jtakos svarstymas, pvz., zinios apie jverti- 
nimo kriterijus. 

* Sociaiiai regiamentuotos normos- vertinimo norma, taikoma 
studentq jvertinimui, kai remiamasi socialiniais palyginimais, t. 
y. atsizvelgiama j jq darbo kokybq lyginant su kitq tos grupes 
darbo kokybe. 

* Sumuojamasis jvertinimas - situacijos jvertinimas duotuoju 
momentu (ypac proceso pabaigoje). 

* Skaidrumas - siuo atveju svarbus protagonistai, sukaupq vi- 
sas zinias apie tikslus ir procesus. 

* Vertybes-\ai\, kas kiekvienam svarbu; (teigiama ar neigiama) 
kokybe, tq svarbumq pavercia norimu ar vertinamu dalyku. 

* Vertybiu konfliktas - atskiras zmogus tuo pat metu turi savo 
asmeniniq vertybiq sistemq (vidines vertybes), o skirtingi zmo- 
nesgali tureti skirtingas vertybes (tarpasmenines vertybes). Skir- 
tingos vertybes gali buti konfliktq, kylanciq jvairiomis veiklos ap- 
linkybemis, priezastis. 


jVADAS 

Zmones, dirbantys dviejq ar keliq socialiniu rysiq san- 
durose, susiduria su daugybe daznai vienas kitam 
priestaraujanciq lukesciu (Nitz, 1993). Atskiri bet kurio 
proceso dalyviai turi skirtingus interesus, kuriq tuo pat 
metu nejmanoma patenkinti. Vertintojams tai sukuria 
dilemas ar konfliktines situacijas. Bandysime isanali- 
zuoti tokiq konfliktiniu situacijij struktur^ ir kai kurias 
joms budingas vertybes; taip pat aptarsime kai kurias 
tipines konfliktines situacijas ir pateiktus pasiulymus. 
Kokybes vertinimo ir vadybos kontekstas, kuriame dis- 
kutuojamos sios problemos, yra Zalcburgo universite- 
te atliktas projektas „Studentai vertina destym^“ (zr. 
darbo grupe ..Evaluation von Lehrveranstaltungen an 
der Universitat Salzburg", 2002, kur pateikiama visa 
informacija ir sqsajos). 


• Fidelity, here reliability; accuracy with which a characte- 
ristic is assessed with a given instrument. 

• Formative evaluation, evaluation which is used to influence 
the process which has been analyzed. 

• Goals in evaluation/ education, describes what someone 
aims at in a given situation. 

• Means in evaluation/ education, describes what someone 
can do to achieve a goal in a given situation 

• Qualitative conflict situation decision situation in which 
the possible actions or the achievement of values are 
mutually exclusive (either-or) 

• Quantitative conflict situation, decision situation in which 
the possible actions can be described as “more or less”, 
with the more one value is satisfied, the less the other can 
be satisfied 

• Responsibility. (1) the force that binds you to your 
obligations and (2) the courses of action demanded by that 
force, with the force being either external (e.g., norm systems 
defined by other people) or internal (e.g., norm systems 
defended by the protagonist, conscience) 

• Responsive evaluation (model)', important values of a 
responsive evaluation process are (1) transparency about 
all steps of the evaluation, (2) participation of all involved 
persons, e.g. partly in important decisions of the evaluation 
process, and (3) consideration of further influences, e.g. 
knowledge about criterion of the evaluation 

• Social reference norm'. Reference norm to evaluate students 
based on social comparisons, i.e. , referring to the quality of 
the work compared to other students of a given group 

• Summative evaluation, evaluation of the situation at a given 
moment in time (particularly at the end of a process) 

• Transparency, here the protagonists have full knowledge 
about the goals and processes 

• Value', something that is important for someone; the quality 
(positive or negative) that renders something desirable or 
valuable 

• Values conflict, one person has many different but simul- 
taneous values (intrapersonal), and between persons there 
may be different values (interpersonal). The different values 
can conflict in varying situations of action. 


INTRODUCTION 

People working in the intersection of two or more social 
networks are confronted with a multitude of expectations 
which often are contradictory (Nitz, 1993). The different 
stakeholders have different interests which cannot be 
fulfilled simultaneously. For evaluators or quality asses- 
sors, this creates dilemma or conflict situations. In the 
following we want to analyze the structure of such con- 
flict situations and some of the typical values involved in 
such situations; further, we want to discuss some proto- 
typical conflict situations and the solutions that were pro- 
posed. The quality assessment and management con- 
text used to discuss these issues is the project “Student 
Ratings of Teaching” at the University of Salzburg (see 
Task Force “Evaluation von Lehrveranstaltungen an der 
Universitat Salzburg”, 2002, for information and links). 
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VERTYBiy KONFLIKTy 
SPRENDIMAS 

Antagonistines ir dilemines 2 situacijos - tai tokios aplinky- 
bes, kuriomis protagonistas turi pasirinkti maziausiai is 
dviejq veiksmij, bet kad ir kq jis ar ji darytq, butinai pazeis 
vienq arba kitq vertybg ar susidurs tam tikros pazeidimq 
normos (zr. Gastager, 1999). Terminas antagonizmasyca 
zodziu „priesingas” ar „priestaravimas” sinonimas. Sis ter- 
minas atsirado dar E. Kanto laikais. Kalbant apie antago- 
nistinius interesus artikslustomis aplinkybemis, kuriomis 
protagonistas turi apsisprqsti, kq daryti (zr. Thiel, 1980, p. 
125ff), du interesai ir l 2 — yra vadinami antagonistiniais, 
jei intereso I, siekimas slepia savyje intereso l 2 prevencijq, 
ir atvirksciai. Tokios antagonistines situacijos gana daz- 
nos; zmonems budinga tureti keletq tuo pat metu egzis- 
tuojanciq vertybiu ar tikslq, kuriuos jie nori patenkinti, ir 
daugeliu atveju sios vertybes buna antagonistiskos. 

Analizuojant tokias antagonistines ir dilemines situa- 
cijas, du dalykus reikia isskirti vertybiu situacijos struktu- 
ros atzvilgiu (zr. 1 lentelg): skirtumq konflikto lygmenyje 
(tikslai ir priemones) ir skirtumq konflikto rusies atzvilgiu 
(kokybinis ar kiekybinis). 


1 

DEALING WITH VALUES 
CONFLICTS 

Antagonistic and dilemma 2 situations are decision situa- 
tions in which the protagonist has to choose between at 
least two actions, but whatever he or she does, he or she 
will necessarily break one or another value, or certain 
norms clash (see Gastager, 1999). The term “antagonism” 
is synonym to “contrary” or “contradiction”. The expres- 
sion “antagonism” goes back to Kant. With respect to 
antagonistic interests or aims in situations in which the 
protagonist has to make a decision what to do (see Thiel, 
1980, 125ff), two interests and l 2 are called antagonistic 
if the pursuit of I, implies the prevention of l 2 and vice 
versa. Such antagonistic situations are frequent; it is typi- 
cal for people to have several simultaneous values or 
goals they want to satisfy, and in many cases these val- 
ues are antagonistic. 

For the analysis of such antagonistic and dilemma situ- 
ations, two distinctions with respect to the structure of the 
situation must be made (see table 1): a distinction on the 
level of the conflict (goals vs. means) and one with regard 
to the type of conflict (qualitative vs. quantitative). 


1 lentele. Konfliktij hierarchine struktura (pagal Patry, 1997a) 
Table 1 . A taxonomy of conflicts (after Patry, 1997a) 




Konflikto rusis / Type of conflict 



Kokybinis /Qualitative 

Kiekybinis / Quantitative 

Konflikto vieta / 

Tikslai / Goals 



Location of the conflict 

Priemones / Means 




Dviejq vertybiq nesuderinamumas gali buti tikslq ar- 
ba pA/e/no/7/i/lygmenyje (Patry, 1997a). Tikslu lygmenyje 
konfliktq galima apibudinti taip: 

1 . Asmuo siekia tikslq G, ir G 2 . 

2. Situacija, apibudinama G, - 1. y. situacija, kuri susi- 
darys, kai bus pasiektas tikslas G,, negali atsirasti tuo 
pat metu kaip ir situacija, apibudinta G 2 . 

Pateikiame ivertinimo situacijos pavyzdj. jvertintojas 
respondentams, atsakinejantiems j atviro tipo anketos 
klausimus, garantavo anonimiskumq, bet administracija 
reikalauja atskleisti kai kuriq teiginiq autorius. Ftespon- 
dentq anonimiskumo islaikymo tikslas nesuderinamas su 
tapatybes atskleidimo tikslu, nes pirmasis yra logiskas 
pastarojo neiginys. 

Antrasis konfliktas, kylantis priemoniq lygmenyje, yra 
toks: specifineje situacijoje zmogus turi du tikslus - G 1 ir 
G 2 . Tam, kad pasiektq G,, reiketq atlikti veiksmq A,, o pa- 
siekti G 2 reiketq atlikti veiksmq A 2 . G, ir G 2 nera vienas 
kitq paneigiantys, bet A t ir A 2 negali buti atliekami tuo pat 
metu. Nera tokio veiksmo (arveiksmq komplekso), kuris 
leistq pasiekti abu tikslus G 1 ir G 2 . Pavyzdziui, tikslai gau- 
ti patikimus duomenis (G^ ir isleisti nedaug pinigq (G 2 ) 
logiskai nera priestaraujantys vienas kitam, bet aukstos 


2 Antagonistiniq ir dileminiq situacijq skirtumq aptarsime ve- 
liau; zr. kokybiniai ir kiekybiniai konftiktai. 


The incompatibility of two values can be either on the 
goat level or on the means level (Patry, 1997a). On the 
goal level, the conflict can be formulated as follows: 

1 . A person aims at the goals G, and G 2 . 

2. The situation described by G 1 - i.e. , the situation 
which would occur if the goal G 1 was achieved - cannot 
occur simultaneously with the situation described by G 2 . 

An example from evaluation. An evaluator has war- 
ranted anonymity to the subjects responding to an open- 
ended questionnaire, but is required by the administra- 
tion to reveal the identity of the author of the different 
statements. The goal of keeping the respondents anony- 
mous is incompatible with the goal of revealing their iden- 
tity, the former being the logical negation of the latter. 

The second conflict - on the means level - is such 
that the person has the two goals G, and G 2 in a particu- 
lar situation. To achieve G,, the action A 1 would be ap- 
propriate, while to achieve G 2 one should do A 2 . G 1 and 
G 2 are not mutually exclusive, but A t and A 2 cannot be 
performed simultaneously. There is no action (or set of 
actions) which would permit to achieve both G 1 and G 2 . 
For instance, the goals of valid data (G,) and of low costs 
(G 2 ) are not logically contradictory, yet providing high 


2 For the distinction between antagonistic and dilemma situ- 
ations, see below: qualitative vs. quantitative conflicts. 
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kokybes duomenq pateikimas (A,) susij^s su daugybe 
darbo (A 2 ), kuris turi buti gerai atlyginamas. 

Norint jsitikinti, ar tai tikslq konfliktas (pirmasis tipas), 
ar ne, reikia pazvelgti j tai, kaip, ar situacijq suderinamu- 
mas apibudinamas pagal iskeltus tikslus; paprastai tokie 
aprasymai is dalies yra tikslq apibudinimas. Kita vertus, 
priemoniq konfliktq metu sprendimus priimantis asmuo 
visq pirma galvoja apie antagonizmq; zmones daznai ma- 
no, kad tuo pat metu nejmanoma pasiekti visq tikslq (A 1 
ir A 2 yra nesuderinami), taciau is tikrqjq tai visiskai jma- 
nomas dalykas, t.y., dar yra veiksmas A 3 apie kurj prota- 
gonistas nepagalvoja ir kuris patenkina G, ir G 2 (pvz., per- 
einant j kitq problemos lygmenj; viena galimybe butu tai- 
kyti metakomunikacijq; pig. Watzlawick ir kt. , 1969). Mi- 
netu atveju galbut butq galima rasti kitq atlygio saltiniu, 
ne tik pinigq: pvz., dali darbo gali atlikti doktoranturos 
studentai rasydami savo daktaro disertacijq. 

Antrasis skirtumas, kurj reikia aptarti, yra kokybiniai ir 
kiekybiniai konfliktai (zr. stulpelius 1 lenteleje). Sis skirtu- 
mas gludi skales, naudojamos elgsenai apibudinti, lygme- 
nyje. Kai kuriuos elgsenos tipus geriausia apibudinti pasi- 
telkiant stebejimo sistemq, kuri suteikia duomenq nomina- 
liajai skalei; siuo atveju subjektas gali rinktis arba/K^ ar A 2 . 
Mes tai vadiname kokybiniais konfiiktais. Pavyzdziui, jver- 
tintojas subjektq vardus atskleidzia arba neatskleidzia. Cia 
kompromiso negali buti, atliekant veiksmq A 1 mes butinai 
pazeisime vertybes, susijusias su A 2 (cia nera svarbu ar tiks- 
lo, ar priemoniq lygmenyje), ir atvirksciai. Tai vadinamoji di- 
iema, tokios rusies konfliktai paprastai sprendziami galimy- 
biq ir vertybiq teorijose (pvz., Feather, 1982; Krampen, 1986) 
ar moraliniq sprendimu studijose (Kohlberg, 1984). 

Kitokios rusies elgsenq galima apibudinti pasitelkus ran- 
gu skalg, t.y. vartojant terminus daugiau ar maziau. Pavyz- 
dziui, anketa gali buti ilgesne ar trumpesne. Tokiu atveju 
konfliktai atspindi elgsenos laipsnj intervale tarp daugiau at 
maziau , tokius konfliktus vadiname kiekybiniais konfiiktais. 
Paprastai subjektai nenukrypsta j krastutinumus. Pavyzdziui, 
kalbant apie anketq ilgumq krastutinumas gali buti vienas 
klausimas; kita vertus, teoriskai nera pasakyta, kokio ilgu- 
mo anketa turetq buti. Taciau paprastai anketos buna nuo 
vieno iki 20 puslapiq. Pasirinkdamas tarn tikrq anketos pus- 
lapiij skaiciq jvertintojas gali siekti kompromiso tarp respon- 
dento motyvacijos (kuo trumpesne anketa, tuo didesne mo- 
tyvacijq jq pildyti) ir gaunamos informacijos (kuo ilgesne 
anketa, tuo didesne tikimybe, kad informacija bus visapu- 
siskesne). Apsisprqsdamas del anketos ilgio jvertintojas gali 
apsibrezti, kurios vertybes svarbesnes, o tai nulems anke- 
tos ilgj. Kad ir kq jvertintojas nusprqstq, jis ar ji negales mak- 
simaliai sutelkti demesio i visus rupimus klausimus, bet tu- 
rns rasti optimalLj variant^ ir kiek galima daugiau atsizvelgti 
j visas vertybes. Negalima nuspr^sti apibendrintai, koks yra 
tas optimalus variantas (pvz., vieno ar desimties puslapiq 
anketa), tai priklauso nuo specifiniq sqlygu ar, kitais zodziais 
tariant, nuo situacijos specifiskumo (Patry, 1991); pavyzdziui, 
imkite paskaitos nutraukimo metod^ (Patry, 1997b), kai pa- 
skaita toliau neskaitoma, o studentai klausinejami apie tai, 
kas vyko per paskutiniqsias penkias minutes ar panasiai; 
tokiu atveju ilgesne nei vieno puslapio anketa negali buti 
naudinga (bet tokj vertinimq galima pakartoti keletq kartq), 
o baigiamoji ( summative ) anketa gali buti ir ilgesne. Priklau- 
somai nuo to, kokiq sqlygu reikia atsakymams (pvz., jei j 
anketos klausimus galima atsakineti visq pamokq, vidutinis 


quality data (A,) requires usually much work (A 2 ) which 
must be paid for. 

For the decision whether there is a goal conflict (first 
type) or not one can look at the compatibility of the situa- 
tions described by the goals; usually, such descriptions are 
part of the goal descriptions. On the other hand, for a con- 
flict of means, the antagonisms are primarily “in the head” 
of a decider; often people think that the simultaneous satis- 
faction of their aims is impossible (A, and A 2 are incompat- 
ible), although it would be quite possible to do so, i.e., there 
is an action A 3 of which the protagonist did not think and 
that satisfies both G 1 and G 2 (e.g., through leaving the level 
of the problem; one possibility would be practicing meta- 
communication, cf. Watzlawick et al., 1969). In the example 
mentioned above, maybe one can find other rewards than 
money - e.g., some of the work is done by graduate stu- 
dents as part of their doctoral dissertation. 

The second distinction refers to qualitative vs. quantita- 
tive coni\\c\.s (columns in table 1). This distinction is rooted 
in the scale level used to describe the behavior. There are 
behavior types which best are described using an observa- 
tion system which provides data on a nominal scale; in this 
case the subject can do either A, orfk 2 . We call this “qualita- 
tive conflicts”. For instance, an evaluator reveals the names 
of the subjects, or he or she does not. A compromise is not 
possible, and doing A 1 will necessarily brake the values 
linked with A 2 (whether on the goal or on the means level is 
not relevant here), and vice versa. This is called “dilemma”, 
and this kind of conflict is typically dealt with in expectancy- 
values theories (e.g., Feather, 1982; Krampen, 1986) or in 
moral judgment studies (Kohlberg, 1984). 

Other types of behavior can be described on an ordinal 
level, i.e., with terms like “more” and “less”. For instance, a 
questionnaire can be longer or shorter. The conflicts then 
are conflicts with respect to the degree of a behavior, be- 
tween “more” and “less”; we call them “quantitative con- 
flicts”. Usually the subjects do not perform the extremes. 
For instance, with respect to the length of the questionnaires, 
at the lower end, one could use only one item; on the other 
end, there is no theoretical upper limit of the length of a 
questionnaire. However, in practice one will use question- 
naires of between one page and 20 pages. By choosing a 
certain number of pages the evaluator can make a com- 
promise between keeping the responders’ motivation (the 
shorter the questionnaire, the better the motivation) and 
getting information (the longer, the more information). By 
deciding on the length of the questionnaire the evaluator 
can decide to what degree to take into account for the dif- 
ferent values associated with questionnaire length. What- 
ever the evaluator decides he or she will not maximally re- 
spect the values at stake, but an optimum is possible in 
which the values are achieved as much as possible. Where 
this optimum lies (e.g., one page or ten pages for a ques- 
tionnaire) cannot be decided in general, but depends on 
the particular conditions or, in other term, is situation spe- 
cific (Patry, 1991); for instance, with the lesson interruption 
method (Patry, 1 997b) in which the lesson in the classroom 
is interrupted at a given time and the students are ques- 
tioned about what happened in the last five minutes or so, a 
questionnaire of more than one page will not work (but one 
can repeat the assessment several times), whereas a 
summative questionnaire can be much longer, depending 
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studentas jqturetu uzpildyti mazdaug per 40 minuciq). Siuo 
atveju negalime kalbeti apie dilemq, nes negalima isskirti 
tik dviejq galimybiu, labiau tiktu kalbeti apie antagonistines 
situacijas. 

Tiek bendro pobudzio ivertinimui, tiek specifiniam jver- 
tinimui labai svarbu, ar konfliktas kokybinis (dilema), ar 
kiekybinis (antagonistinis), nes galimi konflikto sprendi- 
mo budai yra visiskai skirtingi. Tikroje dilemineje situaci- 
joje sprendimo priimti neimanoma atsizvelgiant j visas 
su konfliktu susijusias vertybes ir interesus, o kiekybinio 
konflikto atvejais kompromisas gali buti pasiektas, kai tarn 
tikru laipsniu j tas skirtingas vertybes atsizvelgiama. Smul- 
kiau apie sj skirtumq ir jo implikacijas apskritai skaitykite 
Patry (1997a); siame straipsnyje demesj sutelksime j sio 
skirtumo taikymq jvertinimams ir kokybes vertinimui. 

2 

KONFLIKTUOJANCIOS VERTYBES 
JVERTINIME 

Kaip mineta ankstesniame skyriuje, konfliktinemis aplin- 
kybemis susiduria keletas vertybiq, interesq ir tikslq. Vi- 
same svietimo jvertinimo procese atsakomybe laikoma 
pagrindine vertybe. Smith ir Glass (1987, p.47) siulo at- 
liepiamqjj ( responsive ) jvertinimo modelj, kuriameakcen- 
tuojama atsakomybe, skaidrumas ir teisingumas: „Tam, 
kad atliktq atliepiamqjj jvertinimo, jvertintojas sukuria ste- 
bejimo ir derybq planq. Jis planuoja, kuriuo metu jvairus 
asmenys stebes programos jgyvendinimq” (Smith ir 
Glass, 1987, p. 47). Siuo metu tokie reikalavimai dazniau- 
siai taikomi kiekvienos svietimo situacijos jvertinimui, pvz., 
„atsakomybe uz destymo sritj” yra pagrindinis reikalavi- 
mas “Etiniuose Amerikos svietimo tyrimq asociacijos stan- 
dartuose” (AERA, 1992). Bet jvairiq asmenq, suinteresuo- 
tu institucijos veikla, atsakomybe gali reikalauti netapa- 
ciu veiksmij; tad kitos vertybes, o ne vien atsakomybe, 
tampa svarbios, vadinasi ivertinimas visada sukuria dile- 
mq ar antagonistinp situacijq (zr. Cronbach, 1985). 

Kokios vertybes yra aktualiausios? Metajvertinimo ty- 
rimas apimantis jvertintojo dilemas ir antagonizmus re- 
miantis patyrusiu jvertintojq apklausa (Gastager ir Patry, 
2001), iskelesesias vertybiq sritis: etines, metodines, so- 
cialines ir interaktyviqsias, jstatymines, ekonomines ir as- 
menines; visas jas galite rasti literatures apie jvertinimo 
saltiniuose, pvz., Cronbach (1985, 1987), jvertinimo eti- 
nese rekomendacijose (pvz., AERA 1992) ir jvertinimo 
standartuose (pvz., Deutsche Gesellschaftfur Evaluation, 
2002; Paschon ir Thonhauser, 2002). jvertinimui reikalin- 
gos vertybiq sritys ir vertybiq, kurios isryskeja jvertintoju 
konfliktinese situacijose, pavyzdziai pateikti lenteleje 2. 

1 . Pirmoji sritis - etines vertybes grindziamos etiniais 
pasiteisinimais, apibudintos „Amerikos svietimo tyrimu 
asociacijos etiniuose standartuose" (AERA, 1992). Tokiq 
vertybiq pavyzdziai - sqziningumas ir skaidrumas (pvz., 
jvertinimo priemoniq ir svarbiausiq problemq aiskinimas 
tiriamiesiems). Kita standartine vertybe, minima dauge- 
lyje jvertinimo tyrimq - anonimiskumas; taciau kartais 
zmones pageidauja atverti savo asmeninius duomenis 
viesai (pvz., Shulman, 1990). 

2. Antroji vertybiq sritis susijusi su metodais. bet ku- 
ris vertinimas turi buti patikimas, vertinimo projektas - 


on the conditions of responding (e.g., if a full lesson is re- 
served for answering the questionnaire, the average sub- 
ject should be able to fill it in within about 40 minutes). One 
cannot speak of a dilemma here because there are not only 
two possibilities; it is more appropriate to speak of antago- 
nistic situations. 

Whether a conflict is qualitative (dilemma) or quantita- 
tive (antagonistic) is very important in general and for evalu- 
ation in particular because the possible solutions are com- 
pletely different. In a true dilemma situation a solution tak- 
ing into account all involved values and interests is impos- 
sible, while in situations with a quantitative conflict a com- 
promise can be conceived in which the different values 
can be taken into account to a certain degree. For more 
details on this distinction and its implications in general 
see Patry (1997a); here we will concentrate on its applica- 
tion to evaluations and quality assessment. 

2 

CONFLICTING VALUES IN EVALUATION 

As presented in the previous chapter in conflict situations 
several values, interests or goals clash. In the complete proc- 
ess of an educational evaluation a central value is responsi- 
bility. With Smith and Glass (see 1 987, 47) one should con- 
sider a responsive evaluation model which includes respon- 
sibility, transparency, and fairness: “To do a responsive evalu- 
ation, the evaluator conceives of a plan of observations and 
negotiations. He arranges various persons to observe the 
program” (Smith & Glass, 1987, p. 47). Nowadays in every 
educational evaluation setting these demands are put into 
practice; e.g., “responsibility to the field” is a Guiding Stand- 
ard of the Ethical Standards of the American Educational 
Research Association (AERA, 1 992). But responsibility with 
respect to different stakeholders may mean different actions, 
and other values than responsibility may be important, hence 
evaluation always has a dilemma or antagonistic structure 
(see Cronbach, 1985). 

What values are at stake? In a meta-evaluation study of 
evaluators’ dilemmas and antagonisms based on interviews 
with experienced evaluators (Gastager & Patry, 2001), six 
areas of values were identified: ethical, methodical, social 
and interactive, legal, economic, and personal values; all of 
them can be found at some place in the literature on evalu- 
ation, e.g., in Cronbach (1985, 1987), in the ethical guide- 
lines (e.g., AERA 1992) and standards for evaluation (e.g., 
Deutsche Gesellschaft fur Evaluation, 2002; Paschon & 
Thonhauser, 2002). In table 2, these areas and examples 
for the values involved in evaluators’ conflict situations and 
which are quite typical for the field are listed. 

1 . The first area, ethical values , includes values based 
on ethical justification and refers mostly to the Ethical Stand- 
ards of the American Educational Research Association 
(AERA, 1992). Examples for values are honesty and trans- 
parency (e.g., explaining the instruments and the central 
issues of the evaluation to the investigated persons). A fur- 
ther value is anonymity, which is a standard in most evalua- 
tion studies; however, sometimes the persons prefer to open 
their personal data to the public (e.g. Shulman, 1990). 

2. The second values area concerns the methods', any 
assessment needs to be valid, the design must be such 
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VERTYBig SRITIS / VALUES AREA 

1 Etines vertybes 

1 Ethical values 


2 Metodines vertybes 


2 Methodical values 


3 Socialities ir interaktyvios vertybes 


3 Social and interactive values 


4 jstatymines vertybes 

4 Legal values 

5 Ekonomines vertybes 
5 Economic values 


6 Asmenines vertybes 
6 Personal values 


2 lentele. Vertybiu sritys ir jg pavyzdziai (adaptuota is Gastager ir Patry, 2001) 
Table 2. Values areas with examples (adapted from Gastager & Patry, 2001) 


PAVYZDZIAI / EXAMPLES 

a. Sgziningumas ir skaidrumas; 

b. anonimiskumo apsauga; 

c. teisingumo butinybe; 

d. jvertinamos institucijos noras gauti informacijg; 

e. jvertinamg asmeng apsauga. 

a. honesty/ transparency; 

b. protection of the anonymity; 

c. obligation for truthfulness; 

d. wish for information of the evaluated institution; 

e. protection of the evaluated persons. 

a. objektyvumas, patikimumas, atitinkami metodai; 

b. apreptis; 

c. priemonig tinkamumas; 

d. matavimo objektyvumo siekis; 

e. dalyvavimo laipsnis; 

f. hipotezes patvirtinimas; 

g. jvertintojo profesionalumas. 

a. objectivity/ reliability/ validity, appropriate methods; 

b. bandwidth; 

c. practicability of the instruments; 

d. ideal of an uninfluenced measurement; 

e. degree of participation; 

f. confirm the hypothesis; 

g. professionality of the evaluator. 

a. noras gauti informacijg apie rezultatus; 

b. viesumas; 

c. atskirg jvertinamg grupig tikslg skirtingumas; 

d. intervencijos efektyvumo lukesciai; 

e. jvertintojo ir jvertinamos institucijos reikalavimas dalyvauti ir buti lygiateisiam vertinimo proceso nariui; 

f. jvertinamg asmeng (pvz., lektorig, destytojg) profesionalumas. 

a. wish for information about results; 

b. publicity; 

c. different aims of the different evaluated groups (plurality of aims); 

d. expectations concerning the effectiveness of the intervention; 

e. claim for participation and emancipation between evaluator and evaluated institution; 

f. professionality of the evaluated persons (e.g. lecturer, teacher). 

a. jstatymg leidejo profesionalumas; 

b. noras adaptuoti jvertinamgjj objektg; 

c. noras gauti informacijg apie jvertinamg institucijg; 

d. grjztamojo rysio apie rezultatus reikalavimas; 

e. jvertinimu paremtas sprendimo priemimas. 

a. professionality of the legislator; 

b. wish for adaptation of the evaluation object; 

c. wish for information of the evaluated institution; 

d. claim for feedback concerning the results; 

e. decisions based on evaluation. 

a. kastai; 

b. jvertintojg pastangg isvengimas; 

c. jvertinimo islaidos; 

d. jvertinimo apimtis (pvz., trukme irt.t.); 

e. rezultatg naudingumas. 

a. costs; 

b. avoiding efforts of the evaluators; 

c. expense for the evaluation; 

d. extent of the evaluation (e.g. concerning time etc.); 

e. (practical) usefulness of the results. 

a. jvertinamg asmeng privataus gyvenimo apsauga; 

b. ..tiesiogine pagalba" tiriamiems asmenims; 

c. domejimasis tiriamuoju objektu; 

d. nerimas del rezultatg isspausdinimo; 

e. specifinis grjztamojo rysio poveikis individui. 

a. protection of the private sphere of the evaluated persons; 

b. “direct help” for investigated persons; 

c. interest in the investigation object; 

d. anxiety concerning the publication of the results; 

e. aimed/concrete feedback for the individual. 
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leidziantis atmesti visas susikertancias hipotezes (Camp- 
bell, 1969), duomenu analizejeturi nebuti klaidu irt.t. Ypac 
svarbus klausimas sioje srityje yra jvertintojq ir svietimo 
institucijij profesionalumas. 

3. Trecioji sritis - socia/ines ir interaktyviosios verty- 
bes, apimancios jvairiq suinteresuotq grupiq pageidavi- 
mus ir lukescius. Siq grupiq interesai gali buti visiskai skir- 
tingi, antagonistiniai ar net nesiderinantys. Kai kurios su- 
interesuotos grupes (pvz., administracija) gali noreti gauti 
specifinq informacijq; kiti (pvz., personalas) gali nenoreti 
sios informacijos atskleisti. Tam tikru suinteresuotq gru- 
piq informavimas artimai susijqs su bendru svietimo ins- 
titucijos reikalavimu dometis grjztamuoju rysiu, nes is grjz- 
tamojo rysio gauname informacijq apie tai, kokiq rezulta- 
tq pasiekta jdejus tarn tikrq pastangu (zr. Slavin, 1 986, p. 
374). Taciau tai nereiskia, kad kiekvienasuinteresuotagru- 
pe turetq gauti bet kokiq juos dominanciq informacijq. 
Kitas pavyzdys, atskleidziantis priestaringas suinteresuotq 
grupiq vertybes - daznai pasitaikantis administracijos ke- 
tinimas kq nors keisti, kai dirbantieji nieko keisti nenori. 

4. Ketvirtoji sritis - teisines vertybes, tai visos tos verty- 
bes, kurios pateikiamos jstatymuose ir tie, kuriq pareiga 
priversti tuos jstatymus vykdyti. Siuos paprastai gana api- 
bendrintus reikalavimus galima jvairiai interpretuoti ir tai- 
kyti specifinei situacijai; jvertintojo interpretacijos kartais 
skiriasi nuo jstatymo administratoriq interpretacijq. 

5. Penktoji sritis - ekonomines vertybes, tai vertybes, 
susijusios su jvertinimo finansiniais ir personalo aspek- 
tais ar laiko resursais, skirtingu suinteresuotq grupiq pa- 
stangomis ir islaidomis, isoriniais saltiniais, medziagomis, 
technine ir programine jranga ir 1. 1. 

6. Paskutinioji - sritis - „asmeniniLj vertybiU ‘ sritis; cia 
jeina visos suinteresuotq asmenq, jvertintoju ir 1. 1. asme- 
nines vertybes. Vertybiq pavyzdziai: privataus jvertintojq 
gyvenimo apsauga ar asmeniniai jvertintojo interesai ir t.t. 

Si vertybiq struktura yra sugalvota analitiniams tiks- 
lams. Sriciu eiliskumas, kaip parodyta 2 lenteleje, nereis- 
kia jokio prioriteto ranguojant; kiekvienas atskiras jver- 
tintojas pats nusprendzia, kurioms vertybems teikti pri- 
oritetq duotoje konfliktineje situacijoje. Konfliktai gali is- 
kilti vienoje vertybiq srityje (pvz., tarp metodiniq vertybiq: 
aprepties ir patikimumo dilema; Cronbach, 1970, p. 179 
ft., dar zr. 4.1 poskyris; konfliktai tarp jvairiq suinteresuo- 
tq grupiq, zr. treciqjq vertybiq sritj) arbatarp skritingq ver- 
tybiq sriciq (pvz., metodines ir ekonomines vertybes). 

Visai jmanomas dalykas, kad kai kurios vertybes pri- 
klauso dviem ar kelioms vertybiq sritims. Visos isvardy- 
tos vertybes svarbios, taciau kuriai sriciai teikti pirmeny- 
bq specifineje situacijoje, turi nusprqsti jvertintojas ar kiti 
suinteresuoti asmenys, atsizvelgdami j konkrecias sqly- 
gas; pasak Cronbach (1987, p. 321), „jvertinimas yra me- 
nas”, todel „nera vienintelio geriausio jvertinimo piano". 

3 

STUDENTU DESTYMO REITINGAVIMAS 
ZALCBURGO UNIVERSITETE 

Pries pradedami taikyti strukturinj hierarchijos principq 
(taksonomijq) tyrimui, kurio metu Zalcburgo universiteto 
studentai reitingavo destymq, turetume pirmiausiai api- 
budinti vertinimo projektq ir kontekstq. 1993 m. Austrijo- 


that plausible rival hypotheses are ruled out (Campbell, 
1969), the analysis of the data has to be correct, and so 
on. A particularly important issue is professionalism es- 
pecially of the evaluators and the educational institutions. 

3. The third area is on social and interactive\ia\ues and 
concerns requests and expectancies from the different 
stakeholder groups. These groups may have different and 
antagonistic or mutually exclusive interests. For instance, 
some stakeholders (e.g., authorities) may wish specific 
information; others (e.g., personnel) may not want to re- 
veal it to them. Informing certain stakeholders is closely 
related to the general demand in educational settings for 
feedback, since “feedback” means information on the re- 
sults of one’s efforts (see Slavin, 1986, p. 374). However, 
this does not mean that every stakeholder should get any 
information he or she might be interested in. Another ex- 
ample of opposite stakeholder values which can often be 
found is the intention of the administration to change some- 
thing whereas the employees do not want to change. 

4. Fourth, there is the area of legal values, which includes 
all values provided by the law and by those who are sup- 
posed to enforce it. These usually rather general requirements 
are subject to interpretation and application to the specific 
situation; the evaluator’s interpretation sometimes differs from 
the interpretation by the administrators of the law. 

5. The fifth area, economic values, concerns all the val- 
ues connected with the financial and personnel background 
of the evaluation or the time resources, including the costs, 
the efforts to be spent by the different stakeholders, exter- 
nal resources, material, hard- and software, etc. 

6. Finally, the “ personal values ared’ deals with all 
the intrapersonal values of the stakeholders, the evalua- 
tors, etc. Examples for values are protection of the pri- 
vate sphere of the evaluated persons or the personal in- 
terests of the evaluator in the evaluation, etc. 

This taxonomy of values is conceived for analytical 
purposes. The order of the areas in table 2 is not meant 
to present some priority ranking; rather, it is the respon- 
sibility of the individual evaluator to decide which values 
to prefer in a given conflict situation. Conflicts can be 
within one single values area (e.g., within methodical val- 
ues: the bandwidth-fidelity dilemma, Cronbach, 1970, pp. 
179ff., see below, section 4.1 ; conflicts between different 
stakeholder groups, see the third area) or between val- 
ues areas (e.g., methodical vs. economic values). 

It is quite possible that a certain value fits into two or 
more of the areas. All values mentioned seem quite impor- 
tant, yet which one is to prefer in the particular situation is a 
decision which the evaluator or other stakeholders have to 
take considering the actual conditions; as Cronbach (1 987, 
p. 321) says as general advice: “Evaluation is an art” and 
“(t)here is no single best plan for an evaluation.” 


3 

STUDENT RATINGS OF TEACHING 
AT THE UNIVERSITY OF SALZBURG 

Before applying the taxonomy to the student ratings of 
teaching at the University of Salzburg, this evaluation 
project and its context need to be described. In Aus- 
tria, a new law about the organization of universities 
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je buvo isleistas naujas jstatymas apie universitetu reor- 
ganizavimq (UOG 1993), kuris Zalcburgo universitete bu- 
vo jgyvendintas 1997 m. (siuo metu minetas jstatymas 
jau pakeistas). 18-ame sio jstatymo paragrafe numatyta, 
kaip organizuoti tyrimij ir destymo jvertinimq universite- 
te. Be kitu nurodymq buvo pateiktas ir detalus reglamen- 
tas, kaip organizuoti jvertinimo procesq, kai studentai su- 
darineja destymo reitingus; sie reglamento nurodymai 
apibudinti daug issamiau nei kuri nors kita jvertinimo sri- 
tis (Patry, 2002). Vykdydamas sj jsakymq, Zalcburgo uni- 
versiteto studijq prorektorius inicijavo darbo grupes su- 
formavimq, kurios tikslas buvo tokj jvertinimo atlikti. Jver- 
tinimo koncepcija (Bauman ir kt. , 1999) buvo pateikta uni- 
versiteto bendruomenei viesai aptarti, 2000 vasarq atlik- 
tas zvalgomasis tyrimas, o du visapusio pasiekimLj jverti- 
nimo ciklai vyko 2001 vasarq ir 2001-2002 ziemq. Buvo 
planuoti ir velesni jvertinimai, taciau siuo metu jgyvendi- 
namas naujas jstatymas, kur jvertinimas ne toks griez- 
tas; pastaruoju metu jau akcentuojamos kitos, ne jverti- 
nimo, sritys, o jvertinimo srityje daroma ne daug kas. 

3.1. jVERTINIMO SALYGOS 

Pirmoji darbo grupes uzduotis buvo sukurti koncepcijq 
(Baumann ir kiti, 1 999) ypatingq demesj kreipiant j siuos 
klausimus: 

• Jvertinimas, kai studentai sudarineja destymo rei- 
tingus, neravienintelis destymo jvertinimo elementas. Rei- 
kia jvertinti pazymiu studentams rasymq, testavimo sis- 
temq (ypac atsizvelgiant j sqziningumq ir patikimumq), 
tai, ar studentai sugebejo ismokti tuos dalykus, kuriuos 
turejo ismokti ir t.t. 

• jvertinimo rezultatai yra prasmingi, kai jvertinimas 
naudojamas kaip kokybes vadybos priemone. 

• Nepaisantto, kad jstatyme prioritetasteikiamas su- 
miniam jvertinimui, darbo grupe teike pirmenybq formuo- 
jamajam jvertinimui, t.y. kokybes vertinimui integruoti j 
destymo proceso tobulinimq. 

• Darbo grupe ypac akcentuoja konfidencialumq ir 
duomenq saugumq, nes duomenys - kaip ir daugumoje 
jvertinimo kontekstq atvejq - yra labai subtilus veiksnys; 
viesai neturi buti skelbiama jokia informacija apie atskirq 
kursu ar destytoju jvertinimq, pateikiama gali buti tik api- 
bendrinta informacija. Tai reiskia, kad neketinama suda- 
rineti nei destytojq, nei kursq reitingu, nei atlikti kokio ki- 
to panasaus vertinimo - tai butu nesuderinama su for- 
muojamu sio metodo pobudziu. 

• Praktinio pobudzio priezastys leme tai, kad atsaky- 
mai j anketos klausimus buvo pildomi rastu ir naudoja- 
mas popierinis anketos variantas; kompiuterio anketos 
varianto atsisakyta darant prielaidq, kad tai gali sqlygoti 
zemq respondentg aktyvumo laipsnj ir sukelti kitq nepa- 
togumq. 

Sie principal buvo pristatyti universiteto bendruome- 
nei, bet pries pradedant jvertinimo nebuvo sulaukta di- 
delio demesio del busimo jvertinimo pobudzio, reakcijos 
sulaukta tik pasibaigus jvertinimo procesui. 

Darbo grupe parenge specialaus pobudzio anketo, 
kurios cia neaptarsime (zr. Lengenfelder ir kt., 2001 ; Len- 
genfelder, 2003) ir atliko zvalgomojj tyrimo (n=4887, is 
viso 296 kursai); patikimumui patikrinti buvo atlikta dau- 
gybiniq kintamijjq ir daugybiniu metodq analize (Multit- 
rait-Multimethod-Analysis) (Campbell ir Fiske,1959) (ci- 


(UOG 1993) was decided in 1993 and implemented in 
the University of Salzburg in 1 997 (in the meantime this 
law has been replaced by another one). In paragraph 
18, the law stipulated how evaluations in research and 
teaching must be done in the universities. Among oth- 
ers, detailed regulations about student ratings of teach- 
ing were given, being much more specific than for any 
other domain of evaluation (Patry, 2002). Consequently, 
the vice-rector for teaching at the University of Salz- 
burg initiated a Task Force with the aim to implement 
such an evaluation. A concept for the evaluation 
(Baumann et al . , 1999) was submitted to the Univer- 
sity public for discussion, a pilot study was performed 
in summer 2000 and two series of full assessments 
were done in summer 2001 and in winter 2001-2002. 
Further assessments were planned, however in the 
meantime a new law has been implemented which is 
not so strict with respect to evaluations; currently, the 
focus is on other issues than evaluation, and little is 
being done in this regard. 

3.1 THE CONDITIONS OF THE EVALUATION 
The first activity of the Task Force was to write a concept 
(Baumann et al., 1999) with, in particular, the following 
issues: 

• The student ratings of teaching are not the sole ele- 
ment of evaluation of teaching. One should also evaluate 
the grading and testing system (particularly with respect 
to fairness and validity), whether the students learned 
what they were supposed to learn, etc. 

• An evaluation makes sense only if it leads to conse- 
quences, i.e., when it is used as means for quality man- 
agement. 

• Despite the fact that the law seems to prefer 
summative evaluations, the Task Force preferred forma- 
tive evaluations, i.e. the integration of the assessments 
within a process of improvement. 

• The Task Force put particular emphasis on confi- 
dentiality and data protection, because the data - as in 
most evaluation contexts - are very sensitive; especially 
no information about individual courses or teachers, but 
only aggregated data are to be made accessible to the 
public. This means also that no ranking of teachers or 
courses or of any other kind is intended - such rankings 
would also be incompatible with the formative aim of the 
approach. 

• For practical reasons, a paper-pencil response 
mode for the questionnaire was used; an online ques- 
tionnaire was rejected because of the assumed low re- 
sponse rate and further disadvantages. 

These principles were submitted to the University 
community, but ahead of the evaluation there was little 
interest for a discussion of them - only after the evalua- 
tion was implemented there were reactions. 

The Task Force then developed a questionnaire with 
special features which cannot be discussed here (see 
Lengenfelder et al., 2001 ; Lengenfelder, 2003, for details) 
and performed a pilot study (n=4887 for 296 courses) with 
a Multitrait-Multimethod-Analysis (Campbell & Fiske, 1 959) 
for validation (reported in Lengenfelder et al., in press). 
After some necessary adaptations the questionnaire was 
used in the summer of 2001 (n=20228 for 1176 courses) 
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tuojama is Lengenfelder ir kt. spaudos). Atlikus rei- 
kiamus patobulinimus, anketa buvo platinama 2001 
m. vasarq (n = 20228; is viso 1176 kursai), o po to 
2001-2002 m. ziemq (n = 20062; is viso 1902 kursai) 
visame universitete. Analizes rezultatq cia nepateik- 
sime (juos galima rasti Lengenfelder, 2003), bet atlik- 
sime kai kuriq priestaravimq analizp siekdami iliust- 
ruoti skirtumus, pateiktus 1 ir 2 poskyriuose. 

3.2. SUINTERESUOTIEJI ASMENYS 
IR jg VERTYBES 

| bet kurj jvertinimo kontekstq jtraukiamos jvairios visuo- 
menes grupes, turincios skirtingas vertybes; sios disku- 
sijos tikslu terming suinteresuotieji asmenys vartosime 
laisvai, itraukdami ir tas grupes, kurios paprastai su mi- 
netu terminu nesiejamos. Siuo atveju galime isskirti ke- 
lias suinteresuotq asmenu grupes pagal jg turimas ver- 
tybes (zr. 3 lentel§), tos vertybes smarkiai susipynusios 
su vertybiu sritimis, pateiktomis 2 lenteleje: 

1. Vyriausybe (federaline administracija), butent 
Austrijos federaline svietimo, mokslo ir kulturos ministe- 
rija (atsakinga uz universitetus; http://www. bmbwk.gv.at/ 
start. asp) ir jos atstovai - jstatymai, jg vykdymo taisyk- 
les ir federaliniai administratoriai. Pagal veikiancius jsta- 
tymus, galima isskirti sias vertybes (taip pat zr. 2 lente- 
les vertybiq skiltj - 4): 

a) federaliniai administratoriai siekia patikrinti desty- 
mo universitete efektyvumq ir veiksmingumq ir suteikti 
pagrindq kokybes vadybos sprendimams; be visa kita, 
sprendimai del destytojq (jdarbinimas, kelimas pareigo- 
se, kiti su karjera susijg sprendimai, asmeninio tobuleji- 
mo priemones ir 1. 1.) privalo remtis konkretaus asmens 
destymo reitingavimu, kuri atlieka studentai. Tad jei dvie- 
jq sistemingg vertinimg metu studentai destytojq jvertina 
neigiamai, turi buti imamasi priemonig destymo kokybei 
pagerinti. 

b) jie siekia gauti informacijos apie destymo uni- 
versitete status quo. 

c) jie siekia priversti laikytis jstatymo, pagal kurj 
studentai turi ranguoti destymq. 

2. Universiteto administracija, jskaitant atsakingas 
pareigas universitete uzimancius pareigunus - rekto- 
rig ir dekanus, prorektorig ir prodekanus, kurie atsa- 
kingi uz destymo kokybp, katedrg vedejus, progra- 
mg komitetg vadovus ir t.t. Visi jie kelia savo reikala- 
vimus ir uzduotis, bet jg istekliai gana riboti (zr. 5 skil- 
ties vertybes 2 lenteleje). Jg bendrosios vertybes yra 
sios: 

a) jie nori, kad jvertinimo procesas butq kokybis- 
kas, nors dauguma jg nera nei jvertinimo nei koky- 
bes vadybos specialistai; 

b) jie atlieka daugybg kitg papildomg pareigg (6 
skiltis 2 lenteleje: papildomosios asmenines verty- 
bes). 

jvertinimo ir kokybes vadybos atzvilgiu jie papras- 
tai laikosi siq nuostatg: 

c) nori pagerinti institucijos, uz kuriqjie atsakingi, 
veiklos kokybp; 

d) nori, kad butq laikomasi jstatymg (4 skiltis 2 
lenteleje ir anksciau minetas la). 

3. Destytojai, kuriu kursai yra jvertinami, yra as- 
menys, labiausiai suinteresuoti jvertinimu, kadangi jg 


and in the following winter of 2001 -2002 (n=20062 for 1 902 
courses) for the full University. We are here not reporting 
the analysis of the questionnaire results (which can be seen 
in Lengenfelder, 2003); rather we will perform an analysis 
of some of the antagonisms to illustrate the distinctions 
made above in sections 1 and 2. 

3.2. THE STAKEHOLDERS 
AND THEIR VALUES 

In any evaluation endeavor, many stakeholders with dif- 
ferent values are involved; for the purpose of this discus- 
sion, we take the term “stakeholder” loosely and include 
also some groups which commonly are not listed among 
the stakeholders. In the present case, we can distinguish 
the following stakeholder groups with according values 
(see table 3), the latter having a large overlap with the 
values areas of table 2: 

1) The government (the federal administration) is, in 
particular, the Austrian Federal Ministry for Education, 
Science and Culture (which is responsible for the univer- 
sities; http://www.bmbwk.gv.at/start.asp) and its repre- 
sentatives, and it includes the laws, the rules for its ex- 
ecution, and the federal administrators. According to 
the laws in action, the following values can be identified 
(see also values area 4, table 2): 

a) The federal administrators want to check effective- 
ness and efficiency of University teaching with the aim of 
providing a foundation for quality management decisions; 
among others, decisions on teaching personnel (hiring, 
promotion, other career decisions, personnel develop- 
ment measures, etc.) must be based on the student rat- 
ings of teaching of the respective person. Further, if a 
teacher has been rated negatively by the students for two 
consecutive assessments, measures to increase the 
teaching quality have to be taken. 

b) They want information about the status quo in 
teaching at the universities. 

c) They want to enforce the law with regard to stu- 
dent ratings on teaching. 

2) The University administration includes the officials 
who have a responsible position within the University, 
beginning with the rector and the deans, but also the vice- 
rector and vice-deans responsible for teaching, the heads 
of departments, the heads of program commissions, etc. 
All these people have their requirements and tasks, but 
also only a limited volume of resources (values area 5, 
table 2). They have the following general values: 

a) They want to provide a qualitatively good evalua- 
tion process although most of them are not specialists in 
evaluation or quality management, and 

b) they have many additional duties (area 6 in table 
2: additional intrapersonal values). 

With respect to evaluation and quality management, 
they have typically the following values: 

c) They want to increase the quality of the institution 
for which they are responsible. 

d) They want to enforce the law (area 4 in table 2 as 
well as la above). 

3) The teachers whose courses were evaluated are 
the people most directly concerned by the evaluation 
since the ratings are on their teaching and the decisions 
of the administration have a direct influence on their work 
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SUINTERESUOTIEJI ASMENYS/ 
STAKEHOLDERS 

1 Vyriausybe 

1 Government 

2 Universiteto administracija 

2 University administration 

3 Destytojai 

3 Teachers 


4 Studentai 

4 Students 

5 Moksline bendruomene 

5 Scientific community 

6 jvertintojai 

6 Evaluators 


3 lentele. Suinteresuoti asmenys bei jij (numanomos) vertybes studentams reitinguojant destyma 
Table 3. Stakeholders and their (presumed) values in the student rating of teaching 


NUMANOMOS VERTYBES/ 

PRESUMED VALUES 

a) patikrinti destymo universitete efektyvumq irveiksmingumq siekiant priimti kokybes vadybos sprendimus; 

b) gauti informacijq apie destymo universitete status quo', 

c) priversti laikytis jstatymo del studentq atliekamq destymo reitingu. 

a) check effectiveness and efficiency of University teaching to make quality management decisions; 

b) be informed about the status quo in teaching at the universities; 

c) enforce the law with regard to student ratings on teaching. 

a) jvertinimq atlikti gerai nepaisant ziniij apie jvertinimo principus trukumo; 

b) atlikti kitas pareigas; 

c) pagerinti darbo institucijoje kokybg; 

d) priversti laikytis jstatymq. 

a) provide good evaluation despite lack of knowledge in evaluation principles; 

b) perform other duties; 

c) increase the quality of the institution; 

d) enforce the law. 

a) priimti ir palaikyti jvertinimus; 

b) atitinkamai sqziningi ir, jei galima, teigiami studentq reitingai apie destymq; 

c) jei reitingai neigiami - jokio vieso duomenq naudojimo ir jokig bendrq diskusijq; 

d) aiskus grjztamasis rysys; 

e) sumazinti iki minimumo pastangas ir laikq; 

f) pagerinti destymq nejdedant per daug pastangq. 

a) accept and support evaluations; 

b) appropriate, fair, and if possible positive student ratings; 

c) if the ratings are negative: no general access to the data and no general discussion; 

d) understandable feedback; 

e) minimize effort and time; 

f) improve teaching without putting much effort into it. 

a) jei reikia, pagerinti destymo kokybq; 

b) saziningumas vertinant destymq; 

c) nepadaryti zalos destytojui; 

d) sumazinti iki minimumo pastangas ir laikq; 

e) pakartotinai neatsakineti j tq paciq anketq. 

a) increase teaching quality if necessary; 

b) honesty in rating teaching; 

c) no harm to the teacher; 

d) minimize effort and time; 

e) not responding to the same questionnaire repeatedly, 
zr. 2 vertybiq skiltj, 2 lentele 

See values area 2, table 2 

a) pagerinti destymo universitete kokybg; 

b) jvertinimas kaip tyrejo pastangos; 

c) jstatymus taikyti remiantis savomis interpretacijomis; 

d) sumazinti iki minimumo pastangas ir kastus. 

a) improve the teaching quality at the University; 

b) evaluation as research endeavor; 

c) apply the law according to their own interpretation; 

d) minimize their efforts and costs; 
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darbo reitingai ir administracijos sprendimai daro tie- 
sioginq jtakq jq veiklai ir karjerai. Tai labai heteroge- 
niska grupe; galima isskirti sias jq vertybes: 

a) jie palaiko jvertinimq, ir sutinka, kad jq darbas 
butq ivertinamas. 

b) jie atitinkamai tikisi sqziningo ir, jei jmanoma, tei- 
giamo studentu vertinimo. 

c) jei vertinimas neigiamas, jie nenori, kad apie tai 
butq viesai skelbiama ir diskutuojama. 

d) jie nori sulaukti tokio grjztamojo rysio, kurj jie 
supranta. 

e) jie nenori skirti per daug laiko ir deti per daug 
pastangq jvertinimo procesui. 

f) jie nori tobulinti savo destymq nejdedant perne- 
lyg daug pastangq. 

Galbut sios vertybes nera budingos visiems desty- 
tojams, taciau musq patirtis rodo, kad jos buna gana 
tipiskos. 

4. Studentai \.a\p pat tiesiogiai susijq asmenys, ta- 
ciau kitokiu budu. Jie taip pat sudaro heterogeniskq gru- 
pq, kurios pagrindines vertybes, manytume, yra sios: 

a) jie siekia, kad destymo kokybe geretq; 

b) jie nori jvertinti destymo kokybq kaip galima sq- 
ziningiau; 

c) jie nenori padaryti zalos destytojui; 

d) jie nenori sugaisti daug laiko anketu pildymui; 

e) jie nenori is naujo pildyti tas pacias anketas, ver- 
tindami kelis kursus. 

5. Moksline bendruomene - tai jvertinimo eksper- 
tai, kurie dazniausiai yra jvertintoju kolegos. Jq verty- 
bes atsispindi atlikto jvertinimo kritikoje. Sios grupes 
tikslas - apginti jvertinimo teorinius ir praktinius aspek- 
tus, pasiekti tokj jvertinimq, kuris atitiktu esamq pade- 
tj. Jq vertybes pateiktos metodineje (2 skiltis, 2 lente- 
le) ir etinese skiltyse (1 skiltis). 

6. /vertintojai yra darbo grupes nariai: keturi uni- 
versiteto destytojai, savanoriai, dalyvaujantys siame 
darbe, vienas universiteto darbuotojas (turintis psicho- 
logijos daktaro laipsnj) ir vienas studentas, pasam- 
dyti Siam tikslui. jvertinimas buvo tik dviejq pastarqju 
asmenq pareiga, o du universiteto destytojai taip pat 
buvo ir universiteto administracijos nariai (3 grupe). 
Darbo grupes vertybes aprasytos koncepcijoje (zr. 
auksciau, poskyris 3.1). Be to, svarbios ir sios verty- 
bes: 

a) darbo grupe siekia pagerinti destymo universi- 
tete kokybq; 

b) jvertintojams sis projektas atveria tyrimo galimy- 
bes su visomis is to isplaukianciomis pasekmemis: pa- 
tikrinti hipotezes ir teorijas, gauti apibendrinanciu zi- 
niq ir paskleisti jas mokslinei bendruomenei, pvz., pub- 
likacijq pavidalu; 

c) jstatymai taikomi atsizvelgiant j jvertintoju inter- 
pretacijas, kurios gali skirtis nuo standartiniu interpre- 
taciju, bet islikti tarp galimq interpretacijq; 

d) visi darbo grupes nariai labai uzsiemq, todel sten- 
giasi sumazinti savo pastangas iki minimumo. 

Sios vertybes tera tik spejamos, nes nera atlikta ko- 
kiu nors tyrimu joms jvertinti. Nepaisant to, sqrasas at- 
rodo gana pagrjstas, taciau turime zinoti, kad sis s^ra- 
sas nera galutinis ar issamus, kitos papildomos verty- 
bes taip pat gali buti svarbios. 


and career. This is a very heterogeneous group; never- 
theless some values can be assumed: 

a) They accept and support evaluations, including 
being evaluated. 

b) They want appropriate, fair, and if possible posi- 
tive student ratings. 

c) If the ratings are negative, they do not want them 
to be generally known and discussed. 

d) They want a feedback that they understand. 

e) They do not want to spend much time and effort 
for the evaluation. 

f) They want to improve their teaching without putting 
much more effort into it. 

These values may not be applicable to all teachers, but 
according to our experience they are fairly representative. 

4) The students are also directly concerned, though 
in a very different way. They are also a heterogeneous 
group, whose most important values are, as we assume: 

a) They hope for an increase in the teaching quality if 
necessary. 

b) They want to rate the teaching quality as honestly 
as possible. 

c) They do not want to harm the teacher. 

d) They do not want to spend much time in respond- 
ing to questionnaires. 

e) They do not want to respond to the same ques- 
tionnaire again and again, but with respect to different 
courses. 

5) The scientific community consists in experts in evalu- 
ation, particularly colleagues of the evaluators. Their val- 
ues will become manifest in critiques of the evaluation as 
performed. This group is introduced here to defend the 
values of evaluation research and practice, with the focus 
on achieving an evaluation according to the current state 
of the art. The values are those presented above in the 
methodical (area 2, table 2) and ethical domains (area 1). 

6) The evaluators are the members of the Task Force: 
four University teachers who volunteered for this work, 
one University employee (with a PhD in Psychology), and 
a student engaged for this purpose. Only the latter two 
had the evaluation among their duties, and two of the 
University teachers were also members of the University 
administration (group 3). The values of the Task Force 
are described in the concept (see above, section 3.1). In 
addition the following values are important: 

a) The Task Force wants to improve the teaching qual- 
ity at the University. 

b) It considers the project as research endeavor with 
all its implication: among others, to test hypotheses and 
theories, to gain generalizable knowledge, and to make 
it known to the scientific community, e.g., through publi- 
cations. 

c) It applies the law according to its own interpreta- 
tion which may differ from the standard interpretations 
but remains within the scope of possible ones. 

d) All members of the Task Force have a tight time 
schedule and hence tend to minimize their efforts. 

These values are guesses since no investigation was 
made to assess them. Nevertheless the list seems quite 
reasonable, although one must assume that this list is 
not complete and that additional values might be impor- 
tant. 
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4 

KAI KURIE KONFLIKTAI 

Konfliktai, kuriuos cia aprasysime, atspindi jvairiq sriciq 
jvertintojq nurodytusjvertinimo priestaravimus (Gastager 
ir Patry, 2001), kai kurie is jq jau buvo aptarti teorineje 
literaturoje. Darbo grupe tikejosi susidurti su konfliktais — 
tai bet kurio jvertinimo projekto atributai, kitaip negalima 
tiketis, kad jvertinimas bus sekmingas. Buvo numatyti la- 
biausiai tiketinq problemu sprendimo budai. Bet ne vi- 
sos problemos galejo buti numanytos, kai kurie priesta- 
ravimai pasirode esq sudetingesni nei buvo tiketasi. 

Pirmasis konfliktas tiesiogiai ar netiesiogiai laikomas 
visu kitq priestaravimu, kuriuos cia aptarsime, pagrindu: 
tai priestaravimas tarp ekonominiq vertybiu (5 skiltis; 2 
lentele) ir visu kitq vertybiu; sis dalykas nebuvo netike- 
tas, nes „del biudzeto stokos atliekant apklausq ir vertini- 
mo darbq kai kuriu gerq pasirinkimu visada reikia atsisa- 
kyti” (Cronbach, 1987, p. 322). Pavyzdziui, geras jvertini- 
mas (2 skiltis) yra brangus; tas priestaravimas tarp koky- 
bes ir kastq laikomas kiekybiniu konfliktu priemoniu lyg- 
menyje. Taciau oficialus pareigunai (antroji suinteresuo- 
tu asmenq grupe) daznai nepripazjsta jvertinimo proce- 
duru kastq ir nesuteikia reikalingos finansines paramos. 
Siuo klausimu placiau kalbeti nenorime, taciau reikia pri- 
durti, kad su siuo priestaravimu susiduriama nuolat. 

4.1. APREPTIS PRIES PATIKIMUMA 
Konfliktas tarp aprepties ir patikimumo ypac akivaizdus 
bendravimo srityje (Cronbach, 1987, p. 36f.) ; siuo atveju 
turime galvoje tik siaurq sqvokq, kur teigiama, kad imant 
tarn tikro ilgumo vertinimo priemong (pvz., anketq) susi- 
duriama su priestaravimu tarp sriciq, kurias galima ver- 
tinti, skaiciaus ir vertinimo patikimumo (Cronbach, 1970, 
p. 179). Tai metodiniq vertybiq konfliktas (2 skiltis; 2 len- 
tele), bet ji sukelia poreikis rinktis trumpq priemong, nes 
to reikalauja asmenines vertybes (6 skiltis), kurias gina 
studentai (4 grupes suinteresuoti asmenys, d vertybe) ir 
destytojai (suinteresuoti 3 grupes asmenys; e vertybe). 
Tai jau kiekybinis konfliktas remiantis auksciau pateiktu 
apibrezimu - ir priesingai Cronbach terminologijai - tai 
ne dilema. Sis konfliktas kyla priemoniq lygmenyje: is prin- 
cipo didelio patikimumo tikslas ir daugelis vertinimo sri- 
ciq yra palyginti suderinami dalykai, taciau akivaizdu, kad 
nera priemoniq abu tikslus pasiekti tuo pat metu. 

Kaip teigiama Spearman- Brown formuleje (Cron- 
bach, 1970, p. 161 ir 1 71 ff .) , sis konfliktas atsiranda del 
to, kad anketos patikimumas priklauso nuo jos ilgio. Del 
sios priezasties anketos, kurios paprastai pateikiamos 
studentams, kad jie sudarytu destymo reitingus, yra ga- 
na ilgos (pvz., Rindermann 2001; 51 klausimas; Marsh, 
e.g. 1991 ; 35 klausimai ir 1. 1.). Tokio ilgumo anketa ne- 
butq buvusi priimtina nei studentams, nei destytojams. 
Del sios priezasties darbo grupe nusprende pasitelkti 
kitokio pobudzio tyrimo metodq, islaikydama tq patj pro- 
blemos lygmenj (zr. auksciau, 1 poskyris), ir pakeisti 
anketos klausimq strukturq: vietoj paprastq tradiciniq tei- 
giniq, pvz., „destytojas gerai isaiskina”, darbo grupe pa- 
teike taip vadinamuosius „globalius klausimus”: bendrg 
dimensijos apibudinimg su iliustracijomis ir pavyzdziais 
skliaustuose mazesnemis raidemis. Tokiu atveju visa in- 
formacija pateikiama vienu sakiniu, o ne keliais klausi- 
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SOME CONFLICTS 

The conflicts to be discussed below are quite representa- 
tive for the conflicts in evaluation reported by evaluators 
in different fields (Gastager & Patry, 2001), and some of 
them have already been discussed in the literature. The 
Task Force had anticipated most of them in some way - 
this is a necessity in any evaluation project, otherwise it 
is almost certain that it will fail. And it had also conceived 
some solutions to the most likely problems. But not all 
problems could be anticipated, and some conflicts turned 
out quite different than assumed. 

A first conflict is underlying, directly or indirectly, all 
the other conflicts to be discussed below: the conflict 
between economic values (area 5 in table 2) on one hand 
and many other values on the other hand; this was no 
surprise, since “budgets will always force omission of 
some good choices of directions for inquiry and evalua- 
tive activities” (Cronbach, 1987, p. 322). For instance, 
good evaluation (area 2) is expensive - this antagonism 
between quality and costs is a quantitative conflict on 
the level of the means. However, the officials (stakehol- 
der group 2) often do not recognize the costs of the dif- 
ferent evaluation procedures and hence do not provide 
the necessary financial support. We do not want to go 
into details here, but this issue comes up over and over 
again. 

4.1 BANDWIDTH VS. FIDELITY 
The conflict between bandwidth and fidelity has the broad 
meaning in communication (Cronbach, 1987, pp. 36f.); 
in the present case, we refer only to the narrow concept 
according to which for an assessment instrument (e.g., a 
questionnaire) of a given length there is an antagonism 
between the number of domains which can be assessed 
and the reliability of the assessments (Cronbach, 1970, 
p. 179). This is a conflict within the methodical area (area 
2 in table 2) but caused by the need to use short instru- 
ment as required by the personal values (area 6) defended 
by the students (stakeholder group 4, value d) and the 
teachers (stakeholder group 3, value e). It is a quantita- 
tive conflict, hence according to the definition above - 
and in contrast to Cronbach’s terminology - it is not a 
dilemma. The conflict is on the level of the means: in prin- 
ciple the goals of high reliability and many domains are 
compatible, yet it seems that there is no means to achieve 
both goals simultaneously. 

This conflict is due to the fact that the reliability in- 
creases with its length according to the Spearman-Brown 
formula (Cronbach, 1970, pp. 161 and 1 71 ff.) . For this 
reason the typical questionnaires for the student ratings 
of teaching are quite long (e.g., Rindermann, 2001: 51 
Items; Marsh, e.g. 1991 : 35 items; etc.). A questionnaire 
of this length, however, would not have been accepted 
by the students and teachers. For this reason the Task 
Force decided to use a different approach, leaving the 
level of the problem (see above, section 1 ) and changing 
the structure of the items: Instead of the traditional sim- 
ple items like “the teacher provides good explanations” 
the Task Force used so-called “global items”: General 
descriptions of the dimension and, with smaller charac- 
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mais. Tipiskas pavyzdys galetq buti cia pateikiama fra- 
ze „Studentu motyvavimas (skatinimas, susidomejimo 
pa/aikymas ir t kurios vertinimq reikejo pasirinkti is 
7 skaleje nuo „labai silpnai” iki „labai stipriai”. Is viso 
buvo pateikta 1 7 tokiiq „globaliq“ fraziq, padedanciq jver- 
tinti destytojo darbq skirtingose darbo grupes nustaty- 
tose srityse; toliau buvo duotas bendro pobudzio spren- 
dimas, kurio vertinimo skale nuo „labai gerai“ iki „labai 
blogai“. Apie tyrimo patikimumq jau kalbejome (3.1 po- 
skyryje), jo rezultatai pateikti kitose ataskaitose (Len- 
genfelder ir kt., spaudoje); pasirode, kad tokios prie- 
mones gana patikimos. 

4.2. FORMUOJAMASIS PRIES APIBENDRINANT! 
jVERTINIMA 

Vyriausybe (1 suinteresuota grupe; 3 lentele), savo rei- 
kalavimuose vykdyti jstatymq (c vertybe), atrodo, ypac 
akcentuoja apibendrinamqjj jvertinimq (a ir b vertybes), 
o darbo grupe sutelke demesj j formuojamqjj jvertinimq: 
pareigunai (2 suinteresuota grupe; zr. a vertybe) turejo 
aptarti studentq sudarytus atskiru destytoju reitingus su 
paciais destytojais. Vyriausybe taip pat skatino siq pro- 
cedure (zr. auksciau, 3.2 poskyris, la klausimas), bettik 
su destytojais, kuriq darbas buvo jvertintas blogai; atro- 
do, kad formuojamasis vertinimas buvo taikomas tik 
siems destytojams. Darbo grupes nuomone, reitingai tu- 
retq buti aptariami su visais destytojais. 

Pasirode, kad si procedura sunki ne tik destytojams, 
bet ir asmenims, atsakingiems uz grjztamojo rysio pa- 
teikimq. Akivaizdu, kad nepaisant darbo grupes pastan- 
gq informuoti destytojus apie formuojamqjj metodq, is 
tikrqjq destytojai jo nesuprato; be to, buvo laikomasi 
prielaidos, kad jvertinimas apibendrinamasis; daznai 
(pvz. Bauer, 2001) buvo vadovaujamasi argumentu, kad 
jeigu tam tikras destytojo kursas taikant tam tikrq skalg 
buvo labai zemai jvertintas, tai reiskia, kad toks desty- 
tojas ar destytoja, matyt, yra blogi (tai tarsi charakterio 
savybe ar bruozas, o ne elgsena tam tikroje situacijoje; 
zr., pvz., Jones ir Nisbet, 1971), todel visai nestebina 
tai, kad destytojai gynesi nuo siu insinuacijq. Destytojq 
daromos prielaidos is tikrqjq nebuvo visiskai klaidingos, 
nes asmenys, atsakingi uz grjztamqjj rysj, patys nesu- 
prato sios idejos ir interpretavo neigiamus vertinimus 
panasiai kaip ir destytojai. 

Reikejo issamiau informuoti destytojus ir asmenis, at- 
sakingus uz grjztamqjj rysj, surengti susirinkimus ar net 
organizuoti kursus. Taciau tai buvo nejmanoma del kele- 
to priezasciq. Viena priezastis buvo menkas biudzetas, 
kita priezastis - giliai jsisaknijgs jsitikinimas, kad zmones 
mano zing, kas yra jvertinimas, nors ir prisipazjsta nesq 
sios srities specialistai. Mes visi mokemes ir mokeme ki- 
tus, tai reiskia, kad patyreme kitq jvertinimg ir jvertinome 
patys. Vadinasi, netjeijieirsupranta, kad nera ekspertai, 
vis tiek yra jsitiking, kad ismano kas yra svietimas ir jver- 
tinimas, o siuos ju jsitikinimus pakeisti sunku. Jei desty- 
tojams ir asmenims, atsakingiems uz grjztamgjj rysj, pa- 
teikta informacija apie jvertinimq neatitinka jq nuomones, 
galimas daiktas, kad tas skirtumas bus ignoruojamas, o 
zmogus ir toliau laikysis savo buvusiq jsitikinimq, o gal 
sis asmuo is viso nevaiksto j tokius informacinio pobu- 
dzio susirinkimus, nes nenori leisti laiko tam, kg ir taip 
mano jau zings. 


ters in brackets, some illustrations or examples. Instead 
of using several items, the full information is put into one 
item. A typical item is “Motivating the students (stimula- 
tion, furthering interest, etc.)”, to be rated on a 7-point 
scale from “very weakly” to “very strongly”. Overall, 17 
such global items were used to cover the different domains 
the Task Force had conceived; further, a general judgment 
was included, with the ratings ranging from “very bad” to 
“very good”. The validation study was already mentioned 
above (section 3.1), its results are reported elsewhere 
(Lengenfelder et al. , in press); the instrument proved to be 
quite satisfactory. 

4.2 FORMATIVE VS. SUMMATIVE EVALUATION 
The government (stakeholder group 1 , table 3), in its en- 
forcement of the law (value c), seems to put particular 
emphasis on a summative evaluation (values a and b), 
while the focus of the Task Force was on a formative evalu- 
ation: The officials (stakeholder group 2; see value a) were 
supposed to discuss the results of the ratings with the in- 
dividual teachers. This procedure was also stipulated by 
the government (see above, section 3.2, issue la), but only 
for teachers with low ratings; at least for them a formative 
concept seems to have been aimed at. According to the 
Task Force this should be done for all teachers. 

As it turned out, this kind of procedure is difficult to 
convey both to the teachers and to the officials who were 
supposed to give the feedback. Obviously, despite the 
efforts of the Task Force to inform about the formative 
approach the teachers did not really understand it; in- 
stead, the underlying assumption was that the evalua- 
tion was summative; frequently (e.g., Bauer, 2001) it was 
argued that if a teacher had a low rating in a particular 
course with a particular scale, this meant that he or she 
was supposedly a bad teacher (attribution as disposition 
or trait instead of behavior in a particular situation; see, 
for instance, Jones & Nisbet, 1971), and not surprisingly 
the teachers defended themselves against this insinua- 
tion. Actually the assumption of the teachers were not 
completely false because the officials interpreted the 
negative ratings similarly because they did not understand 
the idea either. 

It would have been required to inform the teachers 
and officials more deeply in meetings or even courses. 
However, this was not possible for several reasons. The 
low budget was one reason, but another is rooted in the 
fact that people believe they know what evaluation is al- 
though they acknowledge that they are not specialists. 
We all have experienced education and have practiced 
education; this means that we have experienced and 
practiced evaluation. In consequence, even if they know 
that they are not experts, they have strong beliefs about 
education and evaluation which are hard to replace. If 
the information about evaluation provided to the teach- 
ers and officials does not comply with the presumed 
knowledge, the difference is likely to be neglected and 
the person sticks to his or her initial beliefs - or maybe 
he or she does not go to such information meetings at all 
because he or she does not want to spend time for some- 
thing he or she thinks to know already. 

There are several conflicts. A first one is between 
summative and formative evaluations - this is on the goal 
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Siuo atveju susiduriame su keliais priestaravimais. Pir- 
masis susidaro tarp apibendrinamojo ir formuojamojo 
jvertinimo - tiksluj lygmenyje, kiekybinis vertinimas (for- 
muojamas ar apibendrinamasis, bet galime apmqstyti ir 
rasti budq jungti tuos du metodus). Antrasis susidaro tarp 
reikalingos informacijos perdavimo ir turimq resursq (kie- 
kybinis; priemones). Treciasis - tarp idejq, kuriq laikesi 
darbo grupe, ir idejq, kuriq laikesi destytojai ir pareigu- 
nai. Priestaravimu priezastis nera anksciau mineta struk- 
turine klasifikacija, tai greiciau jgyvendinimo problema, 
su kuria susiduriame daugelyje sriciq. 

4.3. REGLAMENTUOTOS NORMOS 
Vienas klausimas buvo keliamas nuolat: kq reiskia geras 
destymas? Ar vertinimas ketvertu - teoriskai vidutinis - 
rodo gerq destymq, ar tik vidutiniskq ar net blogq desty- 
mq? Sprendimas apie atskiro objekto vertq priklauso nuo 
to, su kuo vertinamas objektas yra lyginamas. Paprastai 
isskiriami trys nustatytq normq tipai (ypatingais atvejais 
galima prideti ir daugiau tipq, bet siuo atveju pakanka 
trijq): individualios, socialines ir j kriterijus orientuotos ver- 
tinimo normos. Individualiai nustatytos normos reiskia, 
kad individo (siuo atveju destytojo) ankstesnis darbas 
(siuo atveju vieno ar keliq ankstesniu reitingq rezultatai) 
yra lyginami su dabartiniu darbu. Kadangi jvertinimas nera 
laikomas ilgalaikiu nuosekliu tyrimu, tai nejmanoma kon- 
troliuoti visq kintamujq, kurie gali tureti jtakos (pvz., tas 
pats destytojas turetq destyti tq patj kursq panasiems stu- 
dentams panasiomis sqlygomis), todel sis jvertinimas ne- 
gali remtis individualiomis normomis. Taigi arba sociali- 
nes, arba j kriterijus orientuotos normos yra galimos. Pir- 
moji reiskia, kad kazkas yra vertinamas gerai, jei jis yra 
geresnis nei jo kolegos, o j kriterijus orientuotas jvertini- 
mas reiskia, kad individo darbas vertinamas pagal tarn 
tikros instancijos sukurtus kriterijus, ir su siais kriterijais 
individo darbas yra lyginamas. 

jstatymuose (4 vertybiq skiltis, 2 lenteleje) nera ais- 
kiai nurodyta, kokios nustatytos normos turi buti taiko- 
mos. Kaip mineta auksciau (4.2 poskyris), asmenys, 
atsakingi uz grjztamqjj rysi, turi su neigiamus vertini- 
mus gavusiais destytojais aptarti galimq ju darbo pa- 
gerinimq. Sj nurodymq isleidusi ministerija atsisake 
nuostatos remtis socialine norma, kur reikalaujama dis- 
kusijos tuo atveju, kai reitingai yra daug zemesni uz 
vidutinius tarn tikro universiteto destytojq reitingus. Tad 
galima daryti prielaidq, kad remiamasi j kriterijus orien- 
tuota norma. 

Formuojamojo jvertinimo poziuriu (zr. auksciau; 4.2 
poskyris) j kriterijus orientuotu normq taikymas tinkamas, 
jei tik galima juo remtis. Taciau butina tokius kriterijus api- 
brezti, o tai jau yra institucijos ar jos atstovq uzduotis (2 
suinteresuotq asmenq grupe); pavyzdziui, universitetas 
galetq teigti, kad ypac vertinami puikus studentq pasie- 
kimai ar gera destymo atmosfera arba studentq kritinio 
mqstymo ugdymas ir t.t. Galima daryti prielaidq, kad dau- 
guma destytojq siq skirtingq normq pasiekimq taip pat 
laiko priestaringu, kaip ir kiekybinius priemoniq konflik- 
tus: tikslai nebutinai nesuderinami, bet dauguma desty- 
tojq mano, kad vieno is tikslq siekimas gali sukliudyti siekti 
kitq tikslq. 

Kokj kriterijq pasirinkti - tai strateginis universiteto 
sprendimas. Darbo grupe nesijaute esanti kompetentin- 


level and quantitative (formative or summative, but one 
can imagine some ways to combine the two approaches). 
A second is between the necessary information trans- 
mission and the available resources (quantitative; means). 
A third one is between the ideas defended by the Task 
Force and those defended by the officials and teachers. 
This is not a conflict in the sense of the above taxonomy; 
rather it is an implementation problem which is encoun- 
tered frequently in domains. 

4.3 THE REFERENCE NORMS 

One question was asked again and again: What is good? 
Is a rating of “4” - the theoretical average - a sign of 
good teaching, or does it indicate only a medium per- 
formance, or maybe even a bad one? The judgment of a 
single value depends on the reference to which it is com- 
pared. Typically one distinguishes three types of refer- 
ences (for specific cases one can add more types, but 
here these are sufficient): individual, social, and criterion 
oriented reference norms. An individual reference norm 
means that an individual’s (here: a teacher’s) previous 
performance (here: ratings of one or more earlier assess- 
ments) are compared to the present performance. Since 
the evaluation is not conceived as time series study and 
it is impossible to control for all the variables that may 
have an influence (e.g., the same teacher should teach 
the same course with similar students under similar con- 
ditions), the focus of the present evaluation study cannot 
be on the individual norms. Hence, either the social or 
the criterion oriented norm is possible. The former one 
states in that someone is judged to be good if he or she 
is better than his or her peers, whereas the latter refers to 
a criterion defined by some instance and with which the 
performance of the individual is compared. 

In the law (values area 4 in table 2) it is not clearly 
stipulated what type of reference norms are to be used. 
It was mentioned above (section 4.2) that the officials have 
to discuss possible improvements with the teachers who 
have clearly negative ratings. The ministry which issued 
this rule resisted to the temptation to refer to the social 
norm, such as requiring discussions when the ratings are 
clearly below the average ratings of the teachers of the 
respective University. Hence one could assume that the 
underlying reference norm is criterion oriented. 

From the point of view of a formative evaluation (see 
above, section 4.2) the use of criterion oriented norms - 
if available - is appropriate. This requires, however, that 
such criteria are defined, and this is the task of the insti- 
tution or its representatives (stakeholder group 2); for 
instance, the University could say that it puts particular 
emphasis on high achievements of the students, or on a 
good teaching climate, or on educating the students to 
critical thinking, etc. One can assume that most teachers 
regard the attainment of these different norms as antago- 
nistic, as quantitative conflict of the means: the goals are 
not necessarily mutually exclusive, but most teachers 
think that aiming at one of the goals will inhibit aiming at 
the other ones. 

It is a strategic decision of the University what crite- 
rion to prefer. The Task Force did not feel competent to 
make such a decision, rather it suggested to the rector 
and vice-rectors to do so, but so far there has not been 
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ga priimti tokj sprendimq, del to buvo pasiulyta jj priimti 
rektoriui ir prorektoriams, bet kol kas jokio sprendimo 
siuo klausimu nepriimta. Zinoma, tai priespriesa jvertini- 
mo sqvokai, bet asmenims, atsakingiems uz grjztamqjj 
rysj, to negalima perteikti (zr. 2 grupe, a tikslas; 3 lentele: 
ekspertizes stoka). 

SocialiniLj vertinimo normq taikymas formuojamojo 
jvertinimo tikslams neturi prasmes, nes jvertinimo tikslas 
yra pateikti atskirq grjztamqjj rysj atskiriems destytojams 
bei detail stipriq ir silpnq pusiq analizq bei pasiulyti gali- 
mybes tobuleti. Tinkamesnis j kriterijus orientuotas jverti- 
nimas, nors universaliLj normatyvq neturetume taikyti, nes 
atitinkamomis sqlygomis reikalaujama taikyti skirtingus 
kriterijus. Pavyzdziui, net jei visame universitete keliamas 
tikslas pasiekti puikiq laimejimq, vis tiek bus tokiq kursq, 
kaip pavyzdziui, komunikacijos ir socialiniq kompetenci- 
jq jgijimo problemoms skirti kursai, kuriq pagrindinis tiks- 
las ne geri pasiekimai, o socialinis mokymasis, kuriam 
reikalinga gera socialine atmosfera. 

Kitas metodas-j individualius kriterijus orientuotq nor- 
mu taikymas, t.y. destytojo klausiama, kaip jis noretu bu- 
ti vertinamas ar j kokius kriterijus, jo nuomone, vertintojai 
turetq labiau kreipti demesi (del skirtingu tikslq sukelia- 
mu priestaravimq zr. auksciau). Mes si metodq taikeme 
kitame kontekste. Taciau klausti destytojq, kokiems kri- 
terijams jie teiktq pirmenybq, rizikinga, nes tai priesta- 
rautq destytojq tikslui minimalizuoti pastangas (3 grupe, 
e tikslas) ir galetu sukelti pasipriesinimq, jei sis metodas 
bus taikomas dideliu mastu. 

Kadangi nebuvo galima taikyti jokiq j kriterijus orien- 
tuotq normq - nei bendrujq, nei individualiq, tai darbo 
grupe turejo pasitelkti primityvias socialiniq vertinimu 
normas (zr. pavyzdzius apie grjztamqjj rysj www.sbg. 
ac. at/evaluation): destytojas buvo supazindinamas su 
gautais vertinimo rezultatais kiekvienu klausimu (kadangi 
pasiskirstymas buvo labai asimetriskas, tai nusprqsta 
apskaiciuoti medianq, nes vidurkis galejo buti klaidin- 
gas). Toliau buvo nurodytos 20-oji ir 80-oji procentiles. 
Pagal apibrezimq 20 proc. viso universiteto studentq 
(arba pagal tokiq proporcijq parinkti is viso jvertinimui 
atrinktq dalyviq skaiciaus) pateikia zemesnj vertinimq nei 
20-oji procentile, o kiti 20 proc. - virsija 80-qjq procen- 
tilq atitinkamoje skaleje; 60 proc. vertinimas pasiskirs- 
to tarp 20-osios ir 80-osios procentiles. Lyginimas su 
visais universiteto studentais, neatsizvelgiant j tai, koks 
jq skaicius butu kiekviename kurse, koks kurso turinys, 
tipas irt.t., turi buti isdiskutuotas. Reikia pripazinti, kad 
statistiniu poziuriu (2 vertinimo sritis, a pavyzdys, 2 len- 
tele), sis pavyzdys netinkamas (zr. Bauer, 2001); pavyz- 
dziui, skirtingq kursq vertinimui reiketu taikyti skirtingas 
socialines normas. Taciau cia iskyla keli priestaravimai 
(priemoniq, o ne tikslq lygmenyje), j kuriuos reikia atsi- 
zvelgti: 

• Visq pirma, skirtingq socialiniq normq taikymas rei- 
kalauja papildomos informacijos apie kursus. Bauer uz- 
simena apie kursq lankanciq studentq skaiciq; galima pri- 
deti ir kitas kurso charakteristikas, pvz., turinj (pvz., ne- 
teisinga lyginti statistikos kursq su socialiniq kompeten- 
cijq lavinimo uzsiemimais, nes pirmasis yra daug sun- 
kesnis ir ne taip studentq megstamas), bet darbo grupe 
neturejo visq svarbiausiq parametru (2 kokybinis konflik- 
tas vertybiu srityje). 


a decision in this regard. Of course this is opposed to 
the concept of evaluation, but this could not be con- 
veyed to the officials (see group 2, aim a in table 3: lack 
of expertise). 

The use of social reference norms for formative pur- 
poses does not make sense since the aim of the evalua- 
tion is to provide an individual feedback to the individual 
teacher combined with a detailed analysis of the strengths 
and weaknesses and to suggest possibilities of improve- 
ment. The criterion oriented norm is much more appro- 
priate, although maybe one should not use a universal 
norm since the respective conditions may require differ- 
ent criteria. For instance, even if the whole University is 
supposed to aim at high achievements, there are some 
courses like communication and social competence 
trainings in which instead the primary aim is social lear- 
ning requiring a good social atmosphere. 

Another approach could be to use an individual crite- 
rion oriented norm, namely by asking the teacher how 
he or she expects to be rated or what criterion he or she 
wants to pay particular attention to (with the conflict be- 
tween different aims, see above, in mind). We have done 
this in another context; asking the teachers what crite- 
rion they prefer, however, it is against their goal to mini- 
mize effort (group 3, aim e) and will encounter opposi- 
tion if done on a large scale. 

Since no criterion oriented norm was available - nei- 
ther general nor individual norm -, the Task Force had to 
recur to the admittedly primitive social reference norm 
(examples of the feedback are available in www.sbg.ac.at/ 
evaluation): The teacher’s average rating for each item 
on his or her lecture (more precisely: the median, which 
was chosen because the distributions were skewed so 
the average could be misleading) was reported to him 
or her. Further, the 20 lh and 80 th percentiles were indi- 
cated. By definition, 20% of the students of the whole 
University (or if so chosen of a sub-sample) give a rat- 
ing below the 20 th percentile and 20% of them give a 
rating above the 80 th percentile on the particular scale; 
60% have a rating between the 20 lh and the 80 th percen- 
tile. The comparison with all students of the University, 
irrespective of the number of students in the course, 
the content, the type, etc. needs to be argued for. It must 
be acknowledged that from a statistic point of view (va- 
lues area 2, example a in table 2), this approach is inap- 
propriate (see Bauer, 2001); for instance, it would be 
appropriate to use different social norms for different 
types of courses. Flowever, there are several antago- 
nisms (on the level of the means, not of the goals) which 
must be taken into account: 

• First of all, the use of different social norms would 
require additional information about the courses. Bauer 
mentions the number of students participating in the 
course; one could add other characteristics of the 
courses, such as the content (e.g., it is unfair to compare 
courses in statistics with social competence trainings 
because the former are much more difficult and less ap- 
preciated by the students) - but the Task Force did not 
know all the relevant parameters (qualitative conflict within 
values area 2). 

• Second, the explanation of the statistical proce- 
dure to the teachers would have been even more com- 
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• Antra, statistiniu proceduru aiskinimas destytojams 
galejo buti dar sudetingesnis nei si procedura - dauge- 
liui destytojq parinkti metodai ir taip kele problemij (ko- 
kybinis konfliktas 3 suinteresuotu asmenq grupeje, d ver- 
tybes, 3 lentele). 

• Trecia, tokiai analizei pateikti truko ekonominiu re- 
sursq (5 vertybiq skiltis), ir nebuvo galima akcentuoti kitLj 
tikslq, nors jie buvo laikomi dar svarbesniais (kiekybinis 
konfliktas). 

Nevisiskai tinkamq statistiniq metodu naudojimas gali 
buti pateisinamas zvelgiant is etiniq pozicijq (1 vertybiq 
skiltis), nes reglamentuotos normos nera tokios svarbios 
formuojamajam jvertinimui, jos svarbesnes apibendrinan- 
ciam jvertinimui; reglamentuotos normos buvo pateiktos 
informaciniais tikslais ir neakcentuojant minties, kad ze- 
mesnj nei 20-oji procentile jvertinimq gavp destytojai bu- 
vo „blogi” destytojai, o jvertinti auksciau „vidutiniai“ ar 
„geri“ (nors destytojai vjertinimus suvoke butent taip). 20- 
osios procentiles pasirinkimas yra tikrai sutartinis (taip 
buvo pasakyta ir destytojams) ir neturetu buti suvokia- 
mas kaip kritine riba. 

4.4. jVERTINIMO PASEKMES 

jvertinimai prasmingi tik tada, jei vertinimai turi pasekmes. 
Formuojamojo vertinimo koncepcijoje vertinimo pasek- 
me yratiesioginis grjztamasis rysys, kuris sqlygoja prak- 
tin§ kaitq, nes destytojas, isklausps vertinimo rezultatus, 
daro isvadas ir atitinkamai keicia savo tiesioginp veiklq. 
Bet tokius pokycius buvo galima priversti daryti tik tarn 
tikru laipsniu (zr. 4.2 ir 4.3 poskyriai), nes nei universiteto 
administracija, nei destytojai iki galo nesuprato sios kon- 
cepcijos. 

Vietoj formuojamojo atliekant apibendrinamqjj jverti- 
nimq, pasekmiq sulaukiama kitame lygmenyje: uz gerq 
destymq galima apdovanoti, o uz blogq destymq „nu- 
bausti". Kita vertus, vertinimo rezultatais galima remtis 
priimant j darbq, aukstinant pareigose ir atliekant kitus 
su karjera susijusius sprendimus (zr. 1 suinteresuotqjq 
grupe, atikslas, 3 lentele). 

Galimq veiksmq pasirinkimas neigiamai jvertintq des- 
tytojq atzvilgiu („nuobaudos“) gana negausus. Destyto- 
jo is darbo atleisti negalima, uz, kaip manoma, blogq des- 
tymq del teisiniq (4 vertes skiltis, 2 lentele) ir asmeniniu 
priezasciq (6 vertes skiltis), lieka tik keletas kitq priemo- 
niq (ir auksciau mineti formuojamojo vertinimo metodai). 
Socialiniu poziuriu (3 vertes sritis), be to, sutarus, kq ga- 
lima vadinti „geru destymu" (2 vertes skiltis), reiketq ak- 
centuoti tik apdovanojimus uz gerq destymq, o ne nuo- 
baudas uz prast^ destymq. Taciau universiteto pareigu- 
nai nepalaike darbo grupes pasiulymu siuo klausimu, 
jiems pasirode svarbiau nubausti destytojus uz „blog^“ 
destymq. 

Siame kontekste reikia pamineti pakartotinus univer- 
siteto valdzios prasymus paskelbti atskiru jvertinimq re- 
zultatus. Darbo grupe atvirai pasisake pries tokius siuly- 
mus, nes jie priestaravo formuojamojo jvertinimotikslams. 
Siuo poziuriu jstatymas neaiskus ir gali buti interpretuo- 
jamas jvairiai. Taciau nors darbo grupe ir priestaravo, at- 
skiri jvertinimo rezultatai buvo paskelbti vieno fakulteto 
internetiniame puslapyje, kituose fakultetuose rezultatus, 
nors ir sunkiai, bet buvo galima gauti. Tai jau kokybinis 
konfliktas, arba duomenu prieinamumo atzvilgiu, kieky- 


plicated than with the current procedure - and many 
teachers had their problems already with the methods 
chosen (qualitative conflict with stakeholder group 3, 
values d in table 3). 

• Third, the economic resources (values area 5) were 
not sufficient to provide such analyses, or other aims 
which were judged more important would have had to 
be reduced (quantitative conflict). 

The use of statistical approaches which were not ab- 
solutely appropriate is ethically justifiable (values area 1 ) 
because the reference norm has not such an importance 
in formative evaluations as it would have in summative 
evaluations; the reference norms were provided for in- 
formation purposes and not with the idea that those teach- 
ers below the 20 lh percentile were “bad” teachers and 
those above it “medium” or “good” ones (although it was 
perceived so by the teachers) . The choice of the 20 lh per- 
centile was absolutely arbitrary (and this was also com- 
municated to the teachers) and must not be taken as a 
critical mark. 

4.4 CONSEQUENCES OF THE EVALUATION 
Evaluations make sense only if the assessments have 
consequences. In a formative concept the conse- 
quences are in the direct feedback which supposedly 
yields changes in practice since the teacher draws the 
conclusions from the assessments him- or herself and 
acts consequently. But this could be enforced only to 
a little degree (see sections 4.2 and 4.3) because nei- 
ther the officials nor the teachers really understood the 
concept. 

Assuming a summative instead of a formative evalua- 
tion, the consequences can be sought on a different level: 
one can reward good teaching and “punish” bad teach- 
ing. On the other hand, the results of the ratings can be 
used for hiring, promotion, and other career decisions 
for the individual teacher (see stakeholder group 1 , aim 
a in table 3). 

The scope of possible actions with respect to teach- 
ers with negative ratings (“punishment”) is very small. 
One cannot dismiss teachers because of supposedly 
bad teaching for legal (values area 4 in table 2) and 
personal reasons (values area 6), and very few other 
means (besides the formative approaches mentioned 
above) are available. But from the social point of view 
(values area 3), as well as in agreement with what can 
be called “good evaluation” (area 2), the focus should 
be laid on rewarding good teaching instead of punish- 
ing bad one. However, the propositions of the Task Force 
in this regard were not accepted by the officials of the 
University, and for them the focus remained on punish- 
ing “bad” teachers. 

In this context one must also refer to the repeated 
request by officials to publish individual evaluation re- 
sults. The Task Force objected explicitly against such 
propositions because they are in contrast to a formative 
endeavor. In this regard, the law is not clear - it can be 
interpreted in different ways. Against the Task Force’s 
appeal the individual results were published - in one 
Faculty in the internet, the other ones, the results were 
available, yet difficult to get. This is a qualitative conflict - 
or with regard to the degree of accessibility of the data a 
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binis konfliktas: tikslq lygmenyje (viesas rezultatu skelbi- 
mas) asmeniniq vertybiq (6a, 2 lentele) ir to, kaip parei- 
gunai jstatymq interpretavo (4 skiltis, 2 lentele; 2 suinte- 
resuotq asmenq grupes, 3 lentele). 

4.5. STUDENT^ REITINGAI IR (VERTINIMAS 
jstatyme ypac pabreziami studentq sudaryti destymo rei- 
tingai. Diskusijos su destytojais ir asmenimis, atsakingais 
uz grjztamqjj rysj, parade, kad tik studentq reitingai buvo 
laikomi svarbiu dalyku, nepaisant darbo grupes pastan- 
gu pabrezti tai, kad jvertinimui taikant tik destymo reitin- 
gavimq praleidziamos kelios kitos labai svarbios sritys. 
Tiesq sakant tai naturalistinis sofizmas, klaidinga argu- 
mentacija (isvada apie tai, kaip kas nors egzistuoja, pa- 
teikta taip, tarsi is tikrqju taip turetq buti), todel, kad ne- 
galima sakyti, kad jei studentai mano, jog tarn tikras des- 
tymo budas yra geras, jis is tikrujq geras; pavyzdziui, dau- 
gelis destytojq jsitiking, kad studentai negali sprgsti apie 
destytojo kompetentingumq. Todel buvo svarbu rasti ne- 
priklausomus kriterijus (zr. 4.3 poskyrj), norint nustatyti, 
ar destymas geras, ar ne. 

Neabejojame studentq pateiktos informacijos patiki- 
mumu. Teorineje literaturoje aprasyti jvairus efektai, pa- 
vyzdziui, Dr. Fox-efektas (Naftulin, Ware ir Donnelly ci- 
tuota is Marsh ir Ware, 1982: kai aktorius labai israiskin- 
gai deste nesqmoningqturinj ir susilauke aukstq jvertini- 
mq), pazymiq rasymo atlaidumas (Greenwald ir Gillmore, 
1997: studentai vertina geriau, jei tikisi gauti geresnius 
pazymius), lyciq efektas (reitingai priklauso nuoto, ar des- 
tytojas ir studentai tokios pat lyties ar ne). jvairiq studi- 
juojamq dalyku rezultatai sudetingi, pavyzdziui, lyciq efek- 
to atveju: kai kuriuose tyrimuose (pvz., Tatro, 1995), vai- 
kinai studentai geriau jvertino destytojas, o merginos stu- 
dentes destytojus; kituose (aprasyta Smith ir kt., 1994) - 
vaikinai studentai geriau jvertino destytojus, o merginos 
studentes - destytojas. Apskritai sie efektai statistiskai 
reiksmingi, jei tyrimo dalyviq skaicius palyginti gausus, 
bet efekto dydis nezymus ir gali buti ignoruojamas del 
praktiniq sumetimq (Gigliotti ir Buchtel, 1990) ir nekelti 
jokios gresmes patikimumui, jei vertinimas pagrjstas lo- 
gisku teoriniu pagrindu. 

Taciau destymas apima ne vien tai, kq gali reitin- 
guoti studentai. Pavyzdziui, klausimai „Ar tinkama eg- 
zaminq sistema?“, „Ar studentai ismoko tai, kq turejo 
ismokti?“, „Ar tinkamas destymo turinys?" ir t.t. yra 
svarbus, bet j juos negalima atsakyti remiantis destymo 
reitingais. Tai konfliktas tikslq lygmenyje: kokio pobu- 
dzio informacijos klausiama? Siuo atveju viena vertus 
yra jstatymines vertybes, nes jstatymuose akcentuoja- 
mas studentq reitinguojamas destymas, kita vertus pro- 
fesionalumo atliekant jvertinimq kriterijai (2f verte, 2 len- 
tele) ir kokybes vadybos vertybes, kurias turetq ginti 
asmenys, atsakingi uz grjztam^jj rysj (2 suinteresuotq 
asmenq grupe), bet jie to nedaro, nes nera sios srities 
specialistai. 

4.6. DESTYTOjg jTAKA 
TYRIMO REZULTATAMS 

Per daug neteikdami tarn demesio, noretume pamineti ir 
tai, kad kai kurie destytojai bande daryti didesng ar ma- 
zesng jtakq studentq atsakymams. Neoficialiai minimi to- 
kie destytojo poveikio budai: 


quantitative one - on the level of the aims (public avail- 
ability of the results) between the personal values (6a in 
table 2) and the officials’ interpretation of the law (area 4 
in table 2; stakeholder group 2 in table 3). 

4.5 STUDENT RATINGS VS. EVALUATION 

The law puts a strong emphasis on the student ratings of 
teaching. Similarly, the deliberations with the teachers and 
officials showed that only the student ratings were per- 
ceived as relevant, despite the efforts of the Task Force 
to stress that several important areas of teaching are left 
out if only student ratings are used in the evaluation. Ac- 
tually it is a naturalistic fallacy (a conclusion from how 
something is to how something should be) to say that 
since the students think a certain teaching is good it is 
indeed good; for instance, many teachers objected that 
students were not able to judge whether the teacher was 
competent or not. It would be important to have inde- 
pendent criteria (see section 4.3) to say whether a tea- 
ching is good or not. 

We do not doubt the validity of the information by the 
students. Several effects have been discussed in the lite- 
rature such as the Dr. Fox-effect (Naftulin, Ware & 
Donnelly, quoted from Marsh & Ware, 1 982: An actor tea- 
ches very eloquently nonsense content and gets high rat- 
ings), grading leniency (Greenwald & Gillmore, 1997: The 
ratings are better if good grades are expected), gender 
effects (the ratings depend on whether the teacher and 
the students are of the same sex or not). The results of 
the different studies are complex, as can be seen with 
regard to the gender effects: In some studies (e.g., Tatro, 
1995), male students give better ratings to female teach- 
ers and female students to male teachers; in others (re- 
ported by Smith et al., 1994), male students give better 
ratings to male teachers and female students to female 
teachers. In general, these effects are statistically signifi- 
cant given the large samples which are available, but the 
effect sizes are rather small and can be neglected for 
practical purposes (Gigliotti & Buchtel, 1990) and no real 
threat to validity if the assessment is based on a sound 
theoretical framework. 

There is more in teaching, however, than what is rated 
by the students. Questions like “Is the examination sys- 
tem appropriate?”, “Did the students learn what they were 
supposed to learn?”, “Is the teaching content appro- 
priate?”, etc., are important but cannot be answered with 
student ratings. This is a conflict on the level of the aims: 
what kind of information is asked for? In the present case, 
there are, on one side, legal values because the focus of 
the law is on student rating; on the other side there are 
values of professional ity in evaluation (value 2f in table 2) 
and of quality management which should be defended 
by the officials (stakeholder group 2) but actually are not 
because they are not specialists in the domain. 

4.6 CHEATING TEACHERS 

Without attributing too much importance to it, we want to 
mention the fact that some teachers tried to influence 
more or less directly the responses of their students. 
Several ways to do so were reported informally: 

• The teacher watches the students responding to 
the questionnaire. 
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• Destytoja(s) stebi, kaip studentai atsakineja j anke- 
tos klausimus. 

• Destytoja(s) pats sako studentams, kaip atsakyti. 

• Destytoja(s) praso studentu teigiamai jvertinti, sa- 
kydami, kad jq karjera priklauso nuo sio ivertinimo (nors 
taip nera). 

Galima kelti prielaidq, kad tai kokybinis konfliktastarp 
sqziningumo (1 skiltis, a pavyzdys, 2 lentele) irdestytojo 
asmeniniq interesq (asmenines vertybes, 6 skiltis), atsi- 
zvelgiant j tikslus (saliski pries nesaliskus reitingus; taip 
pat zr. 3a; 3 lenteleje). Darbo grupe dejo daug pastangq, 
kad butq isvengta tokio apgaulingo elgesio, bet buvo ne- 
jmanoma pasiekti, kad vertinimo procesq auditorijose or- 
ganizuotq pasaliniai zmones; be to, darbo grupe tikejo 
destytojq sqziningumu ir dare prielaidq, kad jie sudarys 
sqlygas jvertinti teisingai. 

Smulkiau sios problemos nenorime aptarti. Ypac del 
to, kad nera zinoma, ar daznai kartojosi toks apgaulin- 
gas elgesys, nes darbo grupq apie tai pasiekdavo tik ne- 
sisteminga informacija. Reikia pripazinti, kad vis delto re- 
zultatus bandoma paveikti, o toks elgesys kelia rimtq pa- 
vojq jvertinimo rezultatams. Taciau darbo grupe neturejo 
priemoniq, kaip tai apriboti. Akivaizdu, kad a priori nega- 
lima sprgsti apie jvertinamq zmoniu sqziningumq. Galbut 
reiketq prasyti studentus protokoluoti vertinimo process, 
kad butq isvengta tokiq manipuliacijq, bet tai sukeltq pa- 
pildomq konfliktq, susijusiq su studentu ir destytojq pasi- 
tikejimu (1 sritis) ar nepasitikejimo skatinimu (3 sritis), ir 
reikalautq didesniq studentu pastangq (4 suinteresuotq- 
ju asmenq grupe), del to isaugtq vertinimo kastai ir t.t. 
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ISVADOS 

Buvo pateikta jvertinimo priestaravimq strukturine klasifi- 
kacija, kuriq sudaro (1) bendroji konfliktq rusiq tipologija 
(1 lentele), (2) bendrasis vertybiq sriciq sqrasas (2 lente- 
le), ir (3) suinteresuotq asmenq sqrasas (sis sqrasas pa- 
naudotas kaip pavyzdys; zr. 3 lentelg) ir ju vertybes, kurios 
negali buti taikomos bendro pobudzio jvertinimui. Bendro- 
sios vertybiq sritys ir specifies suinteresuotq asmenq ver- 
tybiq sritys smarkiai kertasi, todel siame straipsnyje vienur 
kalbejome apie pirmqsias, kitur apie antrqsias. 

Pasirodo, atliekant dideles apimties jvertinimo, tokj 
kaip ir musq pateiktas, kyla daug antagonistiniq situaci- 
ju. 4 dalyje duotas priestaravimq sqrasas nera isbaigtas, 
galima prideti dar keletq konfliktq, bet daugelis jq nera 
labai svarbus. Cia pateikta strukturine klasifikacija ir me- 
todai atrodo tinkami tokiq konfliktq analizei; taciau si ana- 
lize buvo atlikta ex post facto, nors kai kurie potencialus 
priestaravimai buvo numatyti (zr. darbo grupes Evalua- 
tion von Lehrveranstattungen an der Universitat Salzburg, 
2002), pavyzdziui, aprepties ir patikimumo dilema, kuriai 
sprgsti ieskota specifinio budo irtoks budas isbandytas. 
Buvo numatyti ir ekonominiq vertybiq konfliktai, taciau 
vis del to jie liko milziniski viso vertinimo proceso metu. 
Kiti priestaravimai, pvz., apgaulingas destytojq elgesys 
darbo grupei buvo netiketas. Siuloma strukturine klasifi- 
kacija remtis tada, kai jvertinimas suvokiamas kaip bu- 
das imtis priemoniq isvengti ar sumazinti galimus kon- 
fliktus dar pries pradedant jvertinti. 


• The teacher tells the students directly what to an- 
swer. 

• The teacher asks the students to give positive ra- 
tings since his or her career would depend on the eva- 
luation (which actually was not the case). 

One can assume that this is a qualitative conflict bet- 
ween honesty (area 1, example a in table 2) and per- 
sonal interests of the teachers (personal values, area 6) 
with respect to the aims (biased vs. unbiased ratings; 
see also 3a in table 3). The Task Force had put much 
effort in preventing such cheating behavior, but it was 
impossible to have external people going into the 
classes for the assessment; also the Task Force had 
assumed that the teachers would provide fair as- 
sessment conditions. 

We do not want to discuss this problem into more 
detail. Particularly, it is not known how frequent such a 
cheating was since the Task Force was informed 
unsystematically that such behavior occurred. It must be 
acknowledged that cheating happens and that this is a 
serious threat to the evaluation. But the Task Force had 
no control over such behavior. Obviously fairness of the 
evaluated people cannot be assumed a priori. Maybe one 
should ask the students to write protocols about the as- 
sessment to avoid such manipulations, but this would 
create additional conflicts, involving issues like trust (area 
1) or communicating distrust to both the students and 
the teachers (area 3), increasing the required student ef- 
forts (stakeholder group 4) and the cost of the evalua- 
tion, and so on. 
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CONCLUSIONS 

The taxonomy of antagonisms in evaluation was presented, 
consisting of (1 ) a general typology of conflict types (table 
1), (2) a general list of values areas (table 2), and (3) a list 
of stakeholders and their values specific to the evaluation 
used as example (table 3) and which hence cannot be 
proposed for evaluations in general. There is a large over- 
lap between the general values areas and the specific 
stakeholder values, so in the presentation we referred 
sometimes to the former, sometimes to the latter. 

It turned out that there are many antagonistic situations 
in a large evaluation like the one presented here. The list of 
antagonisms presented in section 4 is not complete at all, 
one could add several conflicts, but most of them are of 
less importance. The taxonomy and the approach presented 
here seemed quite useful for such an analysis of conflicts; 
however, here this analysis was made ex post facto, although 
some of the potential antagonisms were anticipated (see 
Task Force “Evaluation von Lehrveranstaltungen an der 
Universitat Salzburg”, 2002), such as the bandwidth-fidelity 
dilemma for which a specific solution was sought and tested. 
The conflicts with economic values were also anticipated, 
yet remained overwhelming all over the evaluation process. 
Other antagonisms took the Task Force by surprise, such 
as the cheating behavior of teachers. It is suggested that 
the taxonomy should be used when an evaluation is con- 
ceived to take measures to avoid or minimize potential con- 
flicts before the evaluation starts. 
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